🔹 Want to master Azure ETL pipelines? In this video, I’ll walk you through an end-to-end data pipeline in Azure, covering:
✅ Azure Data Factory – Ingesting data from HTTP to ADLS
✅ Azure Databricks – Cleaning & transforming data using PySpark
✅ ADLS Storage – Organizing data using Medallion Architecture (Bronze, Silver, Gold)
✅ Power BI – Connecting and visualizing data
📌 Timestamp -
00:00:00 - Intro
00:01:11 - A] Data Understanding
00:03:21 - B] ETL Pipeline Overview
00:08:38 - C] Understanding Terminology - ETL Process
00:12:46 - D] Understanding Terminology - Medallion Architecture
00:19:24 - E] Understanding Terminology - Required Azure Components
00:27:33 - F] Understanding Terminology - Management Level & Hierarchy
00:30:55 - G] Starting With The Practical Implementation
00:32:08 - 1) Subscription Creation
00:33:20 - 2) Resource Group Creation
00:35:04 - 3) Storage Account Creation
00:38:41 - H] Understanding Terminology - Redundancy Storage Types
01:00:04 - 4) Azure Data Factory Creation
01:03:25 - I] Understanding Terminology - ADF - Copy Activity
01:07:30 - 5) Azure Data Factory - Copy Activity Creation
01:17:33 - 6) Azure Databricks Creation
01:26:45 - 7) Azure Databricks- Compute Creation
01:27:32 - 8) Information Required to Connect Azure Databricks to ADLS
01:31:38 - 9) Azure Databricks- Silver Layer - Cleaning & Transformations
02:00:45 - 10) Azure Databricks - Gold Layer - Transformations
02:24:21 - 11) Connecting ADB to Power BI
02:29:39 - 12) Conclusion
By the end of this video, you’ll understand how to build a real-world ETL pipeline using Azure Data Factory, Databricks, and Power BI! 🚀
📌 Dataset Used: Kaggle – Amazon Prime Movies & TV Shows - https://www.kaggle.com/datasets/shivamb/amazon-prime-movies-and-tv-shows
📌 GitHub Link: https://github.com/RutujaKadam95/AzureETLPipeline-AmazonPrimeDataset
📌 Who is this for? Data Engineers, Data Analysts, and BI Professionals
📌 Suggested Videos:
1) 🚀 Databricks & PySpark Full Course | Master Big Data Processing from Scratch - https://youtu.be/P6dbfkSSmH4?si=DaVrvPRSkd7zDd0c
2) From Data to Business Insights: PySpark on Databricks for Amazon Prime Dataset Analysis 📊🚀 - https://youtu.be/7aZGAf8Luys?si=I54Yv23Y8NdoG5Vt
3) Databricks Journey Begins: Compute, Catalog, Workflows, Data Management, and More! - https://youtu.be/4qreAFJfID4?si=7OreycC3EVSlwPUR
4) Spark & Databricks - Spark Architecture |Memory Management |Application Workflow (Theory) - Part 2 - https://youtu.be/T6CGh-R9C84?si=PCeMTOl_Qc5rMYTK
💬 Got questions? Drop them in the comments! Don’t forget to like, share & subscribe to @DataToCrunch , for more data tutorials! 🔥
💬 Connect with me on Instagram - https://www.instagram.com/datatocrunch?igsh=MTFuMTNuM2N0MjRyMg%3D%3D&utm_source=qr 🔥
#dataengineering #dataanalytics #azure #azuredatafactory #azuredatabricks #powerbi #azuredataengineer #azureportal #databrickstutorial #advancedanalytics #bigdataanalytics #dataanalysis #businessintelligence #businessinsights