This tutorial explores Spark Structured Streaming's Integration with Azure Event Hubs. I explain key terms and concepts of Event Hubs and demonstrate that event payload data from Event Hubs can be ingested, transformed, and written into delta tables. Furthermore, I explain how to write to Event Hubs from Spark Structured Streaming.
Chapters:
00:00- Introduction
02:12- Overview of Azure Event Hubs
07:39- Course setup
14:16- Reading from Event Hubs
18:08- Reading a subset of events from Event Hubs
19:49- Converting event payload to readable format
21:31- Parsing event payload data and writing to Delta table
25:46- Applying time-based aggregations on streams
27:26- Writing streams to Event Hubs
Please subscribe: https://www.youtube.com/channel/UC8d958MxE2t1dr27QNqoOhA
Download demo/exercise notebooks from here:
https://github.com/fazizov/youtube/tree/44a5ae06febe9c473f716e2ca5dbfc8c7cf4b3bb/Data%20engineering%20with%20Databricks/Spark%20Streaming%20Event%20Hubs