Amazon Managed Service for Apache Flink Examples

Example applications in Java, Python, Scala and SQL for Amazon Managed Service for Apache Flink (formerly known as Amazon Kinesis Data Analytics), illustrating various aspects of Apache Flink applications, and simple "getting started" base projects.

Table of Contents

Java Examples

Getting Started

• Getting Started - DataStream API - Skeleton project for a basic Flink Java application using DataStream API
• Getting Started - Table API & SQL - Basic Flink Java application using Table API & SQL with DataStream API

Connectors

• Kinesis Connectors - Examples of Flink Kinesis Connector source and sink (standard and EFO)
• Kinesis Source Deaggregation - Handling Kinesis record deaggregation in the Kinesis source
• Kafka Connectors - Examples of Flink Kafka Connector source and sink
• Kafka Config Providers - Examples of using Kafka Config Providers for secure configuration management
• DynamoDB Stream Source - Reading from DynamoDB Streams as a source
• Kinesis Firehose Sink - Writing data to Amazon Kinesis Data Firehose
• SQS Sink - Writing data to Amazon SQS
• Prometheus Sink - Sending metrics to Prometheus
• Flink CDC - Change Data Capture examples using Flink CDC
• JdbcSink - Writes to a relational database executing upsert statements

Reading and writing files and transactional data lake formats

• Iceberg - Working with Apache Iceberg and Amazon S3 Tables
• S3 Sink - Writing JSON data to Amazon S3
• S3 Avro Sink - Writing Avro format data to Amazon S3
• S3 Avro Source - Reading Avro format data from Amazon S3
• S3 Parquet Sink - Writing Parquet format data to Amazon S3
• S3 Parquet Source - Reading Parquet format data from Amazon S3

Data Formats & Schema Registry

• Avro with Glue Schema Registry - Kinesis - Using Avro format with AWS Glue Schema Registry and Kinesis
• Avro with Glue Schema Registry - Kafka - Using Avro format with AWS Glue Schema Registry and Kafka

Stream Processing Patterns

• Serialization - Serialization of record and state
• Windowing - Time-based window aggregation examples
• Side Outputs - Using side outputs for data routing and filtering
• Async I/O - Asynchronous I/O patterns with retries for external API calls
• Custom Metrics - Creating and publishing custom application metrics
• Fetching credentials from Secrets Manager - Dynamically fetching credentials from AWS Secrets Manager

Utilities

• Fink Data Generator (JSON) - How to use a Flink application as data generator, for functional and load testing.

Python Examples

Getting Started

• Getting Started - Basic PyFlink application Table API & SQL

Handling Python dependencies

• Python Dependencies - Managing Python dependencies in PyFlink applications using requirements.txt
• Packaged Python Dependencies - Managing Python dependencies packaged with the PyFlink application at build time

Connectors

• Datastream Kafka Connector - Using Kafka connector with PyFlink DataStream API
• Kafka Config Providers - Secure configuration management for Kafka in PyFlink
• S3 Sink - Writing data to Amazon S3 using PyFlink
• Firehose Sink - Writing data to Amazon Kinesis Data Firehose
• Iceberg Sink - Writing data to Apache Iceberg tables
• Hudi Sink - Writing data to Apache Hudi tables

Stream Processing Patterns

• Windowing - Time-based window aggregation examples with PyFlink/SQL
• User Defined Functions (UDF) - Creating and using custom functions in PyFlink

Utilities

• Data Generator - Python script for generating sample data to Kinesis Data Streams
• Local Development on Apple Silicon - Setup guide for local development of Flink 1.15 on Apple Silicon Macs (not required with Flink 1.18 or later)

Scala Examples

Getting Started

• Getting Started - DataStream API - Skeleton project for a basic Flink Scala application using DataStream API

Infrastructure & Operations

• Auto Scaling - Custom autoscaler for Amazon Managed Service for Apache Flink
• Scheduled Scaling - Scale applications up and down based on daily time schedules
• Monitoring - Extended CloudWatch Dashboard examples for monitoring applications
• Scripts - Useful shell scripts for interacting with Amazon Managed Service for Apache Flink control plane API

---

Contributing

See Contributing Guidelines for more information.

License Summary

This sample code is made available under the MIT-0 license. See the LICENSE file.