In this video, the ease of use between Decodable and Amazon Managed Service for Apache Flink (MSF) is compared with an end-to-end data engineering project. You will build the projects and experience how both platforms handle real-time streamind data from Kinesis Streams, performing transformations, aggregations, and sinking results into PostgreSQL, S3, Apache Druid, Delta Lake, and Elasticsearch. You will experience how Decodable simplifies pipeline creation with its intuitive interface, versus the more complex setup required for AWS MSF. This comparison will help you decide which platform suits your streaming data needs best!
Timestamps:
00:00 Introduction
1:45 Systems architecture
4:15 Setting up a new Project and Installing Dependencies
11:06 Coding the Sensor Data Producer
25:21 Setting up Kinesis Streams
27:00 Setting up your Decodable Account
29:00 Setting up Connection to AWS Kinesis from Decodable
35:40 Connecting to Kinesis Streaming Sources from Decodable
40:45 Creating Pipelines on Decodable
44:48 Creating Transformation and Aggregation Pipelines on Decodable
59:00 AWS Managed Service for Apache Flink (MSF)
1:00:30 Setting up a new Apache Flink MSF Project
1:19:31 Connecting to Kinesis Streams from Apache Flink
1:36:28 Transforming Kinesis Stream Data with Apache Flink
1:37:35 Writing Data to Postgres Sink with Apache Flink
2:06:15 Packaging Flink Code for AWS MSF
2:15:00 Creating AWS Managed Service for Apache Flink Cluster
2:17:00 Submitting Flink Job to MSF Cluster
2:20:00 Fixing Bugs, Updating Code Snapshots, Repackaging and Resubmission
2:30:10 Wrapping up
2:31:21 Documentation, Resources and Getting help
2:33:00 Outro
Resources and Links:
✅ Free tier sign-up: https://dcdbl.co/3MqqDf7
✅ Quickstart guide to accelerate your real-time ETL, ELT, and streaming initiatives: https://docs.decodable.co/get-started/quickstart
✅ Learn more about Decodable in our Architecture Guide: https://dcdbl.co/3Xpfhyk
Full Flink Source Code: buymeacoffee.com/yusuf.ganiyu/decodable-vs-amazon-managed-service-apache-flink-msf-end-end-de-project
✅ Prefer to read: https://medium.com/towards-data-engineering/decodable-vs-aws-managed-service-for-apache-flink-msf-an-end-to-end-data-engineering-showdown-628ecf0428a8
✨Tags:
Data Engineering Project, Decodable, Amazon Managed Service for Apache Flink, Big Data, Apache Flink, AWS MSF, Data Engineering, Real-time Data Streaming, Kinesis Streams, PostgreSQL, S3, Apache Druid, Delta Lake, Elasticsearch, Data Transformation, Data Aggregation, End-to-End Data Engineering Project, Streaming Data Pipelines, ETL, ELT, Data Integration, AWS Kinesis, Cloud Data Engineering, Data Analytics, Data Processing
✨ Hashtags:
#DataEngineering, #Decodable, #ApacheFlink, #AWSMSF, #StreamingData, #RealTimeData, #KinesisStreams, #ETL, #ELT, #DataPipelines, #BigData, #CloudComputing, #AWS, #DataTransformation, #DataIntegration, #TechTutorial, #CloudServices, #DataAnalytics