In this video I talk to David Jacot who works as a Staff Software Engineer at @Confluent and has been a long time Kafka user, committer and PMC member. We covered how Kafka works internally in great depth.
We use Kafka for various use cases and it works great, but going one level below the abstraction and truly understanding the protocols, techniques and algorithms used is a fun ride.
Chapters:
00:00 Kafka Internals with David Jacot
03:33 Defining Kafka
05:16 Kafka Architecture(s)
11:39 Write Path - Producer sending data
18:35 How does replication work?
25:47 How do we track replication progress?
30:42 Failure Modes: Leader fails
38:18 Consumers: Push vs Pull
40:54 Consumers: How does fetch works?
49:03 Consuming number of bytes vs records
50:50 Optimising consumption
01:00:21 Offset management and choosing partitions
01:09:10 Ending notes
I hope you like this episode and more importantly you learnt some amazing techniques Kafka uses to ensure durability, low latency, simplicity and scalability in its architecture.
Do give this episode a like and share it with your network. Also please subscribe to the channel for content like this.
Other playlists:
Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-
Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17
Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d
Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN
Other episodes:
KsqlDB: https://youtu.be/2yE86P6uD_0
Exactly once semantics: https://youtu.be/twgbAL_EaQw
David's Linkedin: https://www.linkedin.com/in/davidjacot/
our website: www.geeknarrator.com
Cheers,
The GeekNarrator