Azure End-To-End Data Engineering Project | Azure Databricks | | Azure Data Factory | Pyspark Tutorial
🔍 What You'll Learn:
In this extensive 7-hours tutorial, you'll embark on an end-to-end Data Engineering project utilizing cutting-edge technologies like Azure Data Factory, Databricks, and PySpark. You'll explore the Unity Catalog for data governance, work with Delta Lake and Delta Tables for efficient data storage and processing, and implement the Medallion architecture to structure your data pipeline effectively. Throughout the video, you'll encounter real-world scenarios such as Dimensional Data Modeling in Azure Databricks and Slowly Changing Dimensions in Databricks, along with common interview questions to help you prepare for your next job opportunity in the field of data engineering.
Data Source Link : https://github.com/anshlambagit/Azure-DE-Project-Resources/tree/main/Raw%20Data
Databricks Tutorial : https://youtu.be/P5pEeR3xQpI?si=XbQYrYkrVp2jKjfE
PySpark Full Course : https://youtu.be/94w6hPk7nkM?si=Y2EUnifXrZRK3MOj
Azure Data Factory Full Course : https://youtu.be/8zIVOdKyoDA?si=XMgP7hLda_neTHWw
Timestamps:
0:00 Introduction
03:03 Data Architecture (Medallion Architecture)
09:09 Data Ingestion Pipeline Design
14:23 Azure Project Overview
23:38 Data Understanding (API)
29:38 Azure Free Account
32:50 Azure Overview
37:12 Azure Data Lake
41:57 Creating Azure Resources
51:14 Azure SQL Server
57:36 Azure Data Factory Tutorial
1:03:19 Data Ingestion in Azure
1:14:10 Dynamic ETL Pipelines in Azure Data Factory
1:32:53 Incremental loading in Azure Data Factory
1:49:51 Azure Data Factory Real Time Scenarios
2:35:04 Azure Databricks Tutorial
2:48:33 Unity Catalog Azure Databricks
3:01:16 Apache Spark Cluster
3:07:57 Access ADLS Gen2 from Databricks
3:33:33 Data Transformation using PySpark
3:44:37 PySpark Tutorial
5:09:31 Slowly Changing Dimensions using Pyspark
5:45:52 Star Schema Data Model (Fact Table)
5:53:35 Surrogate Keys in Data Warehouse
6:04:55 Databricks Workflows
6:35:16 End to End Azure Data Factory Pipeline
Connect with ME - https://www.linkedin.com/in/ansh-lamba-793681184/
Please Hit the SUBSCRIBE button❤️to support me and my hard work.
⭐Hashtags⭐
#azure #azuresql #databricks #pyspark #dataengineering #dataengineer #dataanalytics #datascience #azureinterviewquestions