Databricks Data Engineering Project for Beginners (FREE Account) | Azure Tutorial

👉 Join the Applied AI Community: https://www.skool.com/applied-ai
👉 Get the resources (Github): https://www.lukejbyrne.com/c/azure-data-eng-e2e-repo

Let's make an End-to-End Data Pipeline in Databricks.

--------------------

📊 Learn Data & AI (50% OFF):
https://datacamp.pxf.io/yqr3jb

🤖 Early access to the Applied AI Community:
https://kickofflabs.com/waitlist/f795d9da

📮 Join the Newsletter:
https://lukejbyrne.com/subscribe

🔗 Follow me on Linkedin:
https://linkedin.com/in/lukejbyrne

--------------------

🏅 RECOMMENDED COURSES:
(All under one subscription)

*Top AI Courses:*
AI Engineer for Developers Course - https://datacamp.pxf.io/gO6WRB
AI Engineer for Data Scientists Course - https://datacamp.pxf.io/XmnX4a
Developing AI Applications Course - https://datacamp.pxf.io/Bny25L

*Top Data Courses:*
Data Engineer in Python - https://datacamp.pxf.io/aOAWNZ
Associate Data Scientist in Python - https://datacamp.pxf.io/AP3Eg1
Data Analyst with Python - https://datacamp.pxf.io/GKVZbk
Associate Data Analyst in SQL - https://datacamp.pxf.io/Z6KyV1

--------------------

0:00 - Introduction
3:47 - What is Databricks and its Architecture
5:33 - Medallion Architecture Explained
10:47 - Setting up Databricks in Azure
19:04 - Access Control and Security
29:04 - Connecting Databricks to Storage
36:07 - Introduction to Databricks Notebooks
44:24 - Setting up Compute Resources
45:54 - Configuring Compute Resources
48:02 - Installing Required Libraries
50:34 - Setting Up Notebooks
53:05 - Bronze Notebook: Fetching Raw Data
1:00:00 - Verifying Data in Bronze Layer
1:02:14 - Silver Notebook: Data Transformation
1:10:09 - Gold Notebook: Advanced Transformations
1:18:01 - Writing to Gold Layer
1:23:19 - Creating a Workflow in Databricks
1:27:25 - Configuring Task Dependencies
1:30:34 - Testing the Workflow
1:33:27 - Conclusion and Next Steps

--------------------

Business inquiries: hello@lukejbyrne.com