Amazon Managed Service for Apache Flink was previously known as Amazon Kinesis Data Analytics for Apache Flink.
Amazon Managed Service for Apache Flink 1.18 (recommended version)
Managed Service for Apache Flink now supports Apache Flink version 1.18.1. Learn about the key new features and changes introduced with Managed Service for Apache Flink support of Apache Flink 1.18.1.
Note
If you are using an earlier supported version of Apache Flink and want to upgrade
your existing applications to Apache Flink 1.18.1, you can do so using in-place
Apache Flink version upgrades. With in-place version upgrades, you retain
application traceability against a single ARN across Apache Flink versions,
including snapshots, logs, metrics, tags, Flink configurations, and more. You can
use this feature in RUNNING and READY state. For more
information, see In-place version upgrades for Apache Flink.
| Supported Features | Description | Apache Flink documentation reference |
|---|---|---|
| Opensearch connector | This connector includes a sink that provides at-least-once guarantees. | github: Opensearch Connector |
| Amazon DynamoDB connector | This connector includes a sink that provides at-least-once guarantees. | Amazon DynamoDB Sink |
| MongoDB connector | This connector includes a source and sink that provide at-least-once guarantees. | MongoDB Connector |
| Decouple Hive with Flink planner |
You can use the Hive dialect directly without the extra JAR swapping. |
FLINK-26603: Decouple Hive with Flink planner |
| Disable WAL in RocksDBWriteBatchWrapper by default |
This provides faster recovery times. |
FLINK-32326: Disable WAL in RocksDBWriteBatchWrapper by default |
|
Improve the watermark aggregation performance when enabling the watermark alignment |
Improves the watermark aggregation performance when enabling the watermark alignment, and adds the related benchmark. |
FLINK-32524: Watermark aggregation performance |
| Make watermark alignment ready for production use | Removes risk of large jobs overloading JobManager |
FLINK-32548: Make watermark alignment ready |
|
Configurable RateLimitingStratey for Async Sink |
RateLimitingStrategy lets you configure the decision of what to scale, when to scale, and how much to scale. |
FLIP-242: Introduce configurable RateLimitingStrategy for Async
Sink |
|
Bulk fetch table and column statistics |
Improved query performance. |
FLIP-247: Bulk fetch of table and column statistics for given partitions |
For the Apache Flink 1.18.1 release documentation, see
Apache Flink 1.18.1 Release Announcement
Changes in Amazon Managed Service for Apache Flink with Apache Flink 1.18
Akka replaced with Pekko
Apache Flink replaced Akka with Pekko in Apache Flink 1.18. This change is fully
supported in Managed Service for Apache Flink from Apache Flink 1.18.1 and later. You don't need to
modify your applications as a result of this change. For more information, see
FLINK-32468:
Replace Akka by Pekko
Support PyFlink Runtime execution in Thread Mode
This Apache Flink change introduces a new execution mode for the Pyflink Runtime framework, Process Mode. Process Mode can now execute Python user-defined functions in the same thread instead of a separate process.
Components
| Component | Version |
|---|---|
| Java | 11 (recommended) |
| Scala |
Since version 1.15, Flink is Scala-agnostic. For reference, MSF Flink 1.18 has been verified against Scala 3.3 (LTS). |
| Managed Service for Apache Flink Flink Runtime (aws-kinesisanalytics-runtime) | 1.2.0 |
| AWS Kinesis Connector (flink-connector-kinesis)[Source] |
4.2.0-1.18 |
| AWS Kinesis Connector (flink-connector-kinesis)[Sink] |
4.2.0-1.18 |
| Apache Beam (Beam applications only) |
As of March 1, 2024, there is no compatible Apache Flink
Runner for Flink 1.18. For more information, see Flink Version Compatibility |
Bug fixes
State compression in Apache Flink 1.18.1
Apache Flink offers optional compression (default: off) for all checkpoints and
savepoints. Apache Flink identified a bug in Flink 1.18.1 where the operator state
couldn't be properly restored when snapshot compression was enabled. This could
result in either data loss or inability to restore from checkpoint. For more
information, see FLINK-34063: When snapshot compression is enabled, rescaling of a source operator leads to some splits getting lost
To resolve this, Amazon Managed Service for Apache Flink has backported the fix that will be included in
future versions of Apache Flink. For more information, see github: Always flush
compression buffers
Known issues
Apache Beam
There is no compatible Apache Flink Runner for Flink 1.18 in Apache Beam. For more
information, see Flink Version Compatibility
Amazon Managed Service for Apache Flink Studio
Studio uses Apache Zeppelin notebooks to provide a single-interface development experience for developing, debugging code, and running Apache Flink stream processing applications. An upgrade is required to Zeppelin’s Flink Interpreter to enable support of Flink 1.18. This work is scheduled with the Zeppelin community and we will update these notes when it is complete. You can continue to use Flink 1.15 with Amazon Managed Service for Apache Flink Studio. For more information, see Creating a Studio notebook.
