DAG dependency in Airflow is a though topic. Or was a though topic. Indeed, with the new version of the TriggerDagRunOperator, in Airflow 2.0 it has never been so easy to create DAG dependencies!
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🏆 Take my course : https://www.udemy.com/course/the-ultimate-hands-on-course-to-master-apache-airflow/?couponCode=BIGPROMO1 to join the legends of Airflow
🚨 My Patreon: https://www.patreon.com/marclamberti to support my work and be friend for life
Materials: https://www.notion.so/The-TriggerDagRunOperator-699b5483f34a42e495183fd028b68267
1. Problem
In Airflow 1.10.X we had multiple choices. Subdags, the ExternalTaskSensor or the TriggerDagRunOperator. One of the most common question is, how can I trigger another DAG B from my DAG A and wait for its completion before executing the next task of DAG A. Well, there wasn't an easy way to that. This only answer was to create a new operator, checking in the metadata database for the DAGRuns and so on. Wasn't easy at all. Well, guess what, it's over! Now in Airflow 2.0, there is a new version of the TriggerDagRunOperator!
2. Solution
The new version of the TriggerDagRunOperator brings two most awaited features, wait_for_completion and reset_dag_run!
3. Benefits
With the new TriggerDagRunOperator you can wait for the completion of the triggered DAG. No need to create your own custom operator anymore. It is as simple as setting the parameter wait_for_completion to true. Moreover, the parameter reset_dag_run allows you to backfill your triggered DAGRuns as well which is absolutely CRITICAL.
Enjoy!