👉 Master AI Agents now: https://www.skool.com/applied-ai-7469
✅ Get the resources (Github) 👉 https://www.lukejbyrne.com/c/azure-data-eng-e2e-repo
This project addresses a critical business need by building a comprehensive data pipeline on Azure. The goal is to extract customer and sales data from an on-premises SQL database, transform it in the cloud, and generate actionable insights through a Power BI dashboard. The dashboard will highlight key performance indicators (KPIs) related to gender distribution and product category sales, allowing stakeholders to filter and analyze data by date, product category, and gender.
Tech Stack: On-Prem SQL DB, Data Factory, Data Lake Gen 2, Databricks, Synapse Analytics, Power BI, Entra ID (Active Directory), Key Vault
--------------------
📊 Learn Data & AI (50% OFF):
https://datacamp.pxf.io/yqr3jb
🤖 Early access to the Applied AI Community:
https://kickofflabs.com/waitlist/f795d9da
📮 Join the Newsletter:
https://lukejbyrne.com/subscribe
🔗 Follow me on Linkedin:
https://linkedin.com/in/lukejbyrne
--------------------
🏅 RECOMMENDED COURSES:
(All under one subscription)
*Top AI Courses:*
AI Engineer for Developers Course - https://datacamp.pxf.io/gO6WRB
AI Engineer for Data Scientists Course - https://datacamp.pxf.io/XmnX4a
Developing AI Applications Course - https://datacamp.pxf.io/Bny25L
*Top Data Courses:*
Data Engineer in Python - https://datacamp.pxf.io/aOAWNZ
Associate Data Scientist in Python - https://datacamp.pxf.io/AP3Eg1
Data Analyst with Python - https://datacamp.pxf.io/GKVZbk
Associate Data Analyst in SQL - https://datacamp.pxf.io/Z6KyV1
--------------------
Further detail on setting up SSMS and SQL: https://youtu.be/z7o5Wju-PZg?si=QnMF0AVB5DxNf182
How to set up PowerBI without work email: https://www.youtube.com/watch?v=9RB5xic9BiY
Mr Ks original vid: https://youtu.be/iQ41WqhHglk?si=drPsOQDhLy-gPIbN
--------------------
Overview
---
00:00:00 - Introduction
00:01:18 - Setting Up the Azure Environment
00:05:45 - SQL Database Configuration
00:10:30 - Overview of Azure Data Lake Storage
SSMS
---
00:15:23 - Configuring Azure Data Factory
00:25:11 - Copying Data from SQL to Data Lake
00:38:05 - Debugging Initial Pipeline Issues
Azure Data Factory
---
00:45:13 - ForEach Activity in Azure Data Factory
00:55:30 - Testing the SQL-to-Bronze Pipeline
01:05:30 - Recap of SQL-to-Bronze Process
01:08:41 - Debugging the Pipeline
01:10:04 - Monitoring Pipeline Runs
01:10:28 - Verifying Data in Bronze Layer
01:11:14 - Completion of the Bronze Data Layer
Databricks
---
01:11:53 - Starting Databricks Configuration
01:14:43 - Creating a Databricks Cluster
01:17:29 - Mounting Data Lake Storage in Databricks
01:23:00 - Transformation in Databricks (Bronze to Silver)
01:33:06 - Automating Data Transformations
01:37:03 - Integrating Databricks with Data Factory
01:41:33 - Pipeline Testing and Monitoring
Synapse Analytics
---
01:45:25 - Loading Data into Synapse Analytics
01:50:07 - Creating Views in Synapse
01:54:40 - Integrating Synapse Views into Data Factory Pipelines
Power BI
---
01:57:57 - Power BI Dashboard Setup
02:03:11 - Building Relationships in Power BI
02:06:48 - Dashboard Filters and Slicers
02:10:01 - Publishing and Sharing Power BI Dashboards
Automation and Active Directory
---
02:13:03 - Automating the Entire Pipeline
02:17:11 - Active Directory (Entra ID) Integration
02:21:33 - Triggering and Monitoring Automated Pipelines
02:29:43 - Final Dashboard Refresh and Validation
Closing
---
02:30:07 - Closing Remarks and Next Steps
--------------------
Business inquiries: [email protected]