This library helps you upgrade your Apache Flink applications, without dropping state, from the Amazon Kinesis Connector (i.e. the package software.amazon.kinesis.connectors) to the Apache Flink Kinesis connector (org.apache.flink.streaming.connectors.kinesis). This will be necessary, for instance, if you are looking to upgrade your Flink Runtime from Flink 1.8/1.11 to Flink 1.13+.

Currently we only support the DataStream API.

What is the Amazon Kinesis Connector?

The Amazon Kinesis Connector (see on Github) was developed to support Enhanced Fan Out (EFO) for Apache Flink 1.8/1.11. It has the Maven coordinates <groupId>software.amazon.kinesis</groupId> <artifactId>amazon-kinesis-connector-flink</artifactId> and has the packaging software.amazon.kinesis.connectors.*.

What is the Apache Kinesis Connector?

The Apache Flink Kinesis Connector (see on GitHub) is supported by the Apache Flink community and the recommended Amazon Kinesis connector.

What is the problem this project solves?

This library allows you to migrate from the Amazon to Apache Kinesis connector while retaining state in the source operator. The operator state includes a map of Kinesis shard and sequence numbers. Without this library your job will fail to start.

Add the library to your project. Example for Maven:

<dependency>
    <groupId>software.amazon.kinesis</groupId>
    <artifactId>amazon-kinesis-connector-flink-state-migrator</artifactId>
    <version>1.0.0</version>
</dependency>

Simply replace FlinkKinesisConsumer with FlinkKinesisConsumerMigrator

env.addSource(new FlinkKinesisConsumerMigrator<>("myInputStream", new SimpleStringSchema(), inputProperties)).uid("my-source");

Important note: The migrator does not work if you don't already have a uid set on your source. A workaround is for you to set the UID Hash of the source manually. See docs. If you don't have a uid set on your consumer, you can use the setUidHash method on the source with FlinkKinesisConsumerMigrator and set the uid hash to the uid hash of the source you're migrating from.

After a successful migration to the Apache Kinesis connector, you can:

• Take a snapshot with your upgraded Flink application + connector
• Switch back to using the Apache FlinkKinesisConsumer
• Remove this library from your dependencies

See CONTRIBUTING for more information.

This project is licensed under the Apache-2.0 License.