Download the notebooks used in this tutorial: https://www.skool.com/microsoft-fabric/classroom/d154aad4?md=7157c442fb334f1ca5dd05d35c63d61c
Looking for Fabric consultancy? Fill in this form: https://forms.office.com/e/h5FaVBwQ7s
Data validation and data quality in general is probably the MOST IMPORTANT thing that need to get right when you're creating analytics solutions in Microsoft Fabric.
Without validated data, what use is that fancy machine learning model, or that fancy Power BI report?
In this video, we look in detail about the why, when and how of data validation in Microsoft Fabric.
I walk through three example notebooks to give you hands-on experience of how to implement at thre different stages in an end-to-end pipeline: incoming table, table data (Spark dataframes) and Power BI semantic models.
Catch up on the Power BI to Microsoft Fabric Transition Guide series here: https://www.youtube.com/playlist?list=PLug2zSFKZmV3eee0W2PJU8XNJbu1dn3-P
Link to the Microsoft Semantic Model Validation article: https://blog.fabric.microsoft.com/en-us/blog/semantic-link-data-validation-using-great-expectations/
#powerbi #microsoftfabric #datavalidation #dataquality
Timeline
0:00 Intro
2:38 Why data validation is so important
3:56 What can go wrong in an analytics pipeline?
8:32 The prolem with Power BI
9:22 The huge opportunity in Fabric
11:45 Three types of data validation in Fabric
12:47 Schema validation overview
14:52 Schema validation with GX demo
23:05 Table/ Spark dataframe validation overview
24:38 DBT
25:33 Spark dataframe validation with GX demo
31:00 Semantic model validation overview
32:45 Semantic model validation wit GX demo
38:50 Wide review of tooling and approaches
43:25 Enterprise-scale data-quality monitoring strategy
46:53 Failure monitoring with Data Activator
48:20 Certifying valid datasets
49:19 Steps to embed data validation in your organisation
51:09 Final words