Kafka - Exactly once semantics with Matthias J. Sax

Hey Everyone, In this episode I and Matthias did a deep dive into the world of exactly once semantics in Kafka. We discussed how Kafka has implemented EOS and how failures of different components are handled. This talk will also provide you different patterns Kafka uses for maintaining reliability, durability and correctness in the system. Enjoy! Chapters: 00:00 Introduction 03:48 What is Exactly once semantics in Kafka? 07:39 Building blocks of Exactly once semantics 10:20 Why is exactly once a hard problem? 12:35 Idempotent producer - how is it implemented? 24:45 What happens when the producer crashes? 32:45 What is a transaction in Kafka world? 33:30 Transactional producer - how it is implemeted? 48:12 What happens when a producer fails? Abort? 51:19 What happens when a broker dies? 55:28 How is high watermark used? 58:48 EOS in stream processing 01:07:25 Topology changes in stream processing 01:18:00 Use cases of EOS I hope you liked the discussion and learnt about EOS in kafka. Please don't forget to hit the like button and subscribe to channel. More and more talks coming soon. Important links: Matthias: https://www.linkedin.com/in/mjsax/ Confluent blog: https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ Enabling EOS in Kafka Streams: https://www.confluent.io/blog/enabling-exactly-once-kafka-streams/ My Discord Server: https://discord.gg/bZ4MnUrg Connect with me on Linkedin: https://www.linkedin.com/in/kaivalya-apte-2217221a/ Join my discord server, where I go LIVE frequently. Cheers, The GeekNarrator