Airflow Dynamic DAGs: The powerful way with Jinja and YAML
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🏆 BECOME A PRO: https://www.udemy.com/course/the-complete-hands-on-course-to-master-apache-airflow/?couponCode=BIGPROMO1
🚨 My Patreon: https://www.patreon.com/marclamberti
Airflow dynamic DAGs can save you a ton of time. As you know, Apache Airflow is written in Python, and DAGs are created via Python scripts. That makes it very flexible and powerful (even complex sometimes). By leveraging Python, you can create DAGs dynamically based on variables, connections, a typical pattern, etc. This very nice way of generating DAGs comes at the price of higher complexity and subtle tricky things that you must know.
Ready?
Lets go!