You can download all the code and the Power BI report file used in this video from the link below:
🔗 https://buymeacoffee.com/mrktalkstech/e/333857
Welcome to an exciting video where we'll build a real-time streaming Data Engineering project using Azure services! This is one of the best ways to get hands-on experience with real-time streaming while understanding architectural decision-making for cost and performance optimization.
In this project, we’ll:
- Set up a real-time weather report system using Weather API
- Implement data ingestion using Azure Databricks and Azure Functions
- Use Event Hub for streaming and Microsoft Fabric for real-time intelligence
- Load data into Kusto DB and create an interactive Power BI dashboard
- Set up real-time alerts for extreme weather using Data Activator
- This project covers everything from data source setup to reporting and alerting, providing hands-on skills in modern data engineering! 💻
TimeStamp:
0:00:00 - Project Overview
0:09:38 - Environment Setup
0:52:13 - Data Ingestion using Azure Databricks (Part 1)
1:27:11 - Data Ingestion using Azure Databricks (Part 2)
2:11:14 - Data Ingestion using Azure Functions
2:52:54 - Cost Analysis and Architectural Decisions
3:09:11 - Event Processing and Loading
3:32:13 - Data Reporting using Microsoft Power BI
4:04:15 - Real-time Alert Configurations
4:26:35 - End to End Pipeline Testing
🔗 Follow along and level up your Data Engineering skills! 💡
#DataEngineering #Azure #RealTimeStreaming #MicrosoftFabric #EndToEndProject #PowerBI
– – – Book a Private One on One Meeting with me (1 Hour) – – –
https://www.buymeacoffee.com/mrktalkstech/e/166354
– – – Express your encouragement by brewing up a cup of support for me – – –
https://www.buymeacoffee.com/mrktalkstech
– – – Other useful playlist: – – –
1. Microsoft Fabric Playlist: https://www.youtube.com/playlist?list=PLrG_BXEk3kXybedCIBBI4lmaIbtbn7MdM
2. Azure General Topics Playlist: https://www.youtube.com/playlist?list=PLrG_BXEk3kXxv0IEASoJRTHuRq_DUqrjR
3. Azure Data Factory Playlist: https://www.youtube.com/playlist?list=PLrG_BXEk3kXwTClTt3_28CMz2dZoaFhKD
4. Databricks CICD Playlist: https://www.youtube.com/playlist?list=PLrG_BXEk3kXzMLDAgRbDsbIKvhoWsu-Gq
5. Azure Databricks Playlist: https://www.youtube.com/playlist?list=PLrG_BXEk3kXznRvTJXwmazGCvTSxdCMsN
6. Azure End to End Project Playlist: https://www.youtube.com/playlist?list=PLrG_BXEk3kXx6KE4nBmhf6QwSHMbznP2W
7. End to End Azure Data Engineering Project: https://youtu.be/iQ41WqhHglk
– – – Let’s Connect: – – –
Email: [email protected]
Instagram: mrk_talkstech
– – – About me: – – –
Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN ABOUT THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"
I will be creating playlist which covers the below topics (with DEMO)
1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)
After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,
1. Data Engineer
2. Data Analyst
3. Data Scientist
Can't wait to help people with my videos.
– – – Support me: – – –
Please Subscribe: https://www.youtube.com/@mr.ktalkstech