Last week saw the General Availability of dynamic functionality for Databricks Workflows, in the form of the parameterized ForEach activity, but what does that mean? And why should we care?
For a long time we've been using external orchestration tools whenever things had to be flexible, metadata driven or simply at a large scale - but with these changes we have a cheap, flexible way of achieving some pretty complex orchestration inside of Databricks itself!
In this video Simon looks at a quick pattern for running a medallion-style lake processing routine, all driven by some JSON metadata!
For the release announcements, see: https://www.databricks.com/blog/streamlining-repetitive-tasks-databricks-workflows