This video will focus on how to use Azure Data Factory CI/CD Pipelines and promote ETL Pipelines from Dev environment to UAT then production. How to create an Azure DevOps pipeline to deploy Azure Data Factory pipelines, link services, triggers, datasets, etc to several Azure Data Factories.
Learn a complete tutorial for creating Azure Data Factory CI/CD pipelines with Git enabled that use different branches for developing new features for ETL pipelines and other Azure Data Factory components.
This will cover, creating three Azure Data Factories, two Azure Storage accounts, Azure Release pipeline with Azure Artifacts based on Git repo and deploying Azure Data Factory ARM templates files to another environment.
It will show how to develop all the Azure Data Factory components in the feature branch. Then how to raise a pull request to merge those changes with the master branch. After that publish Azure Data Factory development which will create adf_publish branch which will include the ARM template that describes the current Azure Data Factory Environment. And by using this ARM template, we can deploy it to UAT and Production environments.
Commands:
Stop trigger
---------------
-armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" -ResourceGroupName "[RG name]" -DataFactoryName "[Target ADF Env]" -predeployment $true -deleteDeployment $false
Start trigger
-------------
-armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" -ResourceGroupName "[RG name]" -DataFactoryName "[Target ADF Env]" -predeployment $true -deleteDeployment $true
Stop trigger
---------------
-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $false
Start trigger
-------------
-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $true
-factoryName "[target ADF name]" -AzureBlobStorage1_connectionString "" -AzureBlobStorage2_connectionString ""
-factoryName [target ADF name] -AzureBlobStorage1_connectionString "" -AzureBlobStorage2_connectionString ""
-factoryName "ADF-Radwan-UAT" -AzureBlobStorage1_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan1;AccountKey=xx;EndpointSuffix=core.windows.net" -AzureBlobStorage2_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan2;AccountKey=xxx;EndpointSuffix=core.windows.net"
-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $false
About the Author
-----------------------------
Video:
https://www.youtube.com/watch?v=3Na_5DjFb8g
Blog:
https://mohamedradwan.com
Linked-in
https://eg.linkedin.com/in/mohamedahmedradwan
GitHub
https://github.com/MohamedRadwan-DevOps
Twitter:
https://twitter.com/mradwan06
# DevOps #Azure #Azure_Data_Factory_tutorial #Azure_training #Azure_DevOps