Azure End-To-End Data Engineering Project | Azure DevOps | Azure Data Factory | Databricks | PySpark
In this video, I guide you through a beginner-friendly, end-to-end Azure Data Engineering project using the latest technologies like Azure Data Factory, Databricks, PySpark, Azure Data Lake, and CI/CD with Azure DevOps. Plus, I cover some Azure Data Engineer interview questions!
Timestamps:
0:00 Introduction
08:41 Azure Project Architecture
15:31 Data Understanding
22:04 Azure Free Account
26:08 Azure Fundamentals
32:23 Creating Azure Data Lake
42:26 CI CD in Azure
57:46 Azure DevOps Tutorial
1:07:57 Azure Data Factory Git Configuration
1:17:03 Data Ingestion from Github in Azure Data Factory
1:30:56 Parametrized ETL Pipelines using ForEach Activity
2:01:52 Azure Data Factory Real Time Scenarios
2:32:07 Creating Pull Request in Azure DevOps
2:40:42 ARM templates in Azure
2:45:00 Azure Databricks Tutorial
3:02:00 Databricks with Unity Catalog
3:20:54 Data Ingestion using PySpark in Databricks
3:42:00 Dynamic Notebooks using Databricks Utilities
3:58:11 Data Orchestration using Databricks Workflows
4:16:12 PySpark Tutorial
4:33:04 Big Data Transformation using PySpark
4:56:43 Databricks Delta Live Tables
5:11:27 Spark Structured Streaming
5:26:32 Change Data Capture (CDC) in Databricks
5:36:30 Apache Spark Cluster
5:38:56 End To End ETL Pipeline in Databricks
WATCH MY PREVIOUS VIDEOS
➡️Azure End To End Data Project
https://youtu.be/uc-u_juRg-w?si=sO_CavjnhNaSnZri
➡️Databricks Tutorial
https://youtu.be/7pee6_Sq3VY?si=pPBrTnE2R8PhhHu5
➡️PySpark Full Course
https://youtu.be/94w6hPk7nkM?si=i3u_8ZWkGf6fiGyh
REPOSITORY
➡️GitHub
https://github.com/anshlambagit/AzureProjectWithCICD
➡️Original Dataset
https://www.kaggle.com/datasets/piterfm/paris-2024-olympic-summer-games
=======================================
Join this channel to SUPPORT MY HARDWORK:
https://www.youtube.com/channel/UCu7lQE-L5gzt8aD7zuuufjw/join
=======================================
Connect with ME
LinkedIn - https://www.linkedin.com/in/ansh-lamba-793681184/
Telegram - https://t.me/anshlambadatafam
=========================
For COLLABORATION 👇
anshlambacollab@gmail.com
=========================
⭐Hashtags⭐
#azure #databricks #dataengineering #pyspark #azuredatafactory