Amazon Managed Apache Flink Integration
A fully managed service to help you process streaming data in real time using standard SQL. With Site24x7's Managed Apache Flink (formerly known as Amazon Kinesis Data Analytics) integration you can collect, visualize and alert on metrics to effectively manage your Managed Apache Flink application.
Setup and configuration
- If you haven't done it already, enable access to your AWS resource by creating Site24x7 as an IAM user or by creating a cross-account IAM role between your account and Site24x7's AWS account. Learn more.
- Next, In the Integrate AWS Account page, please make sure the Managed Apache Flink checkbox is selected in the Services to be discovered field. Learn more.
Policies and Permissions
Please ensure the following read level actions are present in the policy document assigned to the Site24x7 IAM entity. Learn more.
- "kinesisanalyticsv2:ListApplications",
- "kinesisanalyticsv2:ListTagsForResource",
- "kinesisanalyticsv2:DescribeApplication
Polling frequency
Site24x7 queries the CloudWatch API as per the poll frequency set (1 minute to a day), to collect performance metrics. Learn more.
Licensing
Each Managed Apache Flink monitor is considered a basic monitor.
IT Automations
You can add automations for the AWS services supported by Site24x7. Log in to Site24x7 and go to Admin > IT Automation Templates (+) > Add Automation Templates. Once automations are added, you can schedule them to be executed one after the other.
You can now start or stop data analytics application using Managed Apache Flink automations.
To view data
- Log in to the Site24x7 console, click on AWS > Monitored AWS account and select Managed Apache Flink from the drop down.
- From the list of monitored applications, choose the application for which you want to access metrics for.
- The various graphs applicable for your analytics application are displayed in the Application Details tab.
Supported metrics
The following metrics are collected:
| Metric name | Description | Statistic | Unit |
|---|---|---|---|
| Bytes | Measures the number of bytes read (per input stream) or written (per output stream). | Sum | Bytes |
| Millis behind latest | Measures how far behind from the current time an application is reading from the streaming source. | Average | Milliseconds |
| Records | Measures the number of records read (per input stream) or written (per output stream). | Sum | Count |
| Success | Gets incremented by 1 for every successfull delivery to the destination, configured for your application. | Sum | Count |
| Input processing duration | Measures the time taken for each Lambda function invocation performed by Managed Apache Flink. | Average | Milliseconds |
| Input Processing OkRecords | Measures the number of records returned by a Lambda function that were marked with Ok status. | Sum | Count |
| Input processing OkBytes | Measures the sum of bytes of the records returned by a Lambda function that were marked with Ok status. | Sum | Bytes |
| Input processing dropped records | Measures the time taken from when the complete fragment data is received and archived. | Average | Milliseconds |
| Input processing processing failed records | Measures the number of records returned by a Lambda function that were marked with ProcessingFailed status. | Sum | Count |
| Input processing success | Measures the number of successful Lambda invocations by Managed Apache Flink. | Sum | Count |
| Lambda delivery OkRecords | Measures the number of records returned by a Lambda function that were marked with Ok status. | Sum | Count |
| LambdaDelivery. delivery failedRecords | Measures the number of records returned by a Lambda function that were marked with DeliveryFailed status. | Sum | Count |
| LambdaDelivery.Duration | The time taken for each Lambda function invocation performed by Managed Apache Flink. | Average | Milliseconds |
| CpuUtilization | The overall percentage of CPU utilization across task managers (TaskManager JVM process). | Average | Percentage |
| ContainerCPUUtilization | The overall percentage of CPU utilization across task-manager containers (full container, not just JVM). | Average | Percentage |
| ContainerMemoryUtilization | The overall percentage of memory utilization across task-manager containers (working-set memory). | Average | Percentage |
| ContainerDiskUtilization | The overall percentage of disk utilization across task-manager containers (root-volume filesystem). | Average | Percentage |
| HeapMemoryUtilization | The overall heap-memory utilization across task managers. | Average | Percentage |
| ManagedMemoryUtilization | The Flink managed-memory used divided by the total (Flink 1.13 only). | Average | Percentage |
| ManagedMemoryUsed | The amount of Flink managed memory currently used (RocksDB state backend; Flink 1.13 only). | Maximum | Bytes, MB |
| ManagedMemoryTotal | The total Flink managed memory (Flink 1.13 only). | Maximum | Bytes, MB |
| LastCheckpointSize | The total size of the last checkpoint (running storage utilization). | Maximum | Bytes, MB |
| LastCheckpointDuration | The time taken to complete the last checkpoint. | Maximum | ms |
| NumberOfFailedCheckpoints | The number of times checkpointing has failed. | Sum | Count |
| NumRecordsIn | The total records this application has received. | Sum | Count |
| NumRecordsInPerSecond | The records received per second. | Average | Count/sec |
| NumRecordsOut | The total records this application has emitted. | Sum | Count |
| NumRecordsOutPerSecond | The records emitted per second. | Average | Count/sec |
| NumLateRecordsDropped | The records dropped due to late arrival. | Sum | Count |
| CurrentInputWatermark | The last input watermark this application received (min of last received watermarks). | Maximum | ms |
| CurrentOutputWatermark | The last output watermark this application emitted. | Maximum | ms |
| OldGenerationGCCount | The total old-generation GC operations across all task managers. | Sum | Count |
| OldGenerationGCTime | The total time spent in old-generation GC. | Sum | ms |
| ThreadsCount | The total live threads used by the application. | Maximum | Count |
| FailingTime | Time the application has spent in a failing state (Flink 2.2+; replaces part of downtime ). | Maximum | ms |
| CancellingTime | Time spent in a cancelling state (Flink 2.2+). | Maximum | ms |
| RestartingTime | Time spent in a restarting state (Flink 2.2+). | Maximum | ms |
| RunningTime | Time the application has been running uninterrupted (Flink 2.2+; replaces uptime ). | Maximum | ms |
| JobmanagerFileDescriptorsMax | The maximum file descriptors available to the JobManager. | Maximum | Count |
| JobmanagerFileDescriptorsOpen | The current open file descriptors for the JobManager. | Maximum | Count |
| TaskmanagerFileDescriptorsMax | The maximum file descriptors available to each TaskManager. | Maximum | Count |
| TaskmanagerFileDescriptorsOpen | The current open file descriptors for each TaskManager. | Maximum | Count |
| Attribute | Description |
|---|---|
| Application Name | Displays the name for the Managed Apache Flink application. |
| Application ARN | Shows the Amazon Resource Name for the application. |
| Application Status | Shows the current status for the application. |
| Version ID | Displays the current application version ID. |
| Creation Time | Shows the time stamp when the application was created. |
| Last Updated Time | Shows the time stamp when the application was last updated. |
| Application Code | Shows the application code that you provided to perform data analysis. |
| Application Description | Shows the description of the application. |
| Logging Option ID | Displays the ID of the CloudWatch logging option description. |