MLOps Community Meetup #94! Last week, we talked to Kevin Kho, Orchestrating Machine Learning Workflows with Prefect, co-hosted by Ben Epstein
//Abstract
Prefect is a general-purpose workflow orchestration management system that enables users to build, run, and monitor data pipelines at scale.
In this talk, Kevin shows how to orchestrate a full ML hyperparameter grid search pipeline over a Dask cluster. Other tools included will be Pandera, Tune, and Evidently AI.
// Bio
Kevin Kho is an Open Source Community Engineer at Prefect, an open-source workflow orchestration management system. Previously, he was a data scientist for four years working in the energy and HR spaces. Outside of work, he is a contributor for Fugue, an abstraction layer for Pandas, Spark, and Dask. He also organizes the Orlando Machine Learning and Data Science Meetup.
// Related links
Prefect Orion Docs: https://orion-docs_prefect.io
Announcing Orion: https://www.prefect.io/blog/announcing-prefect-orion/
----------- ✌️Connect With Us ✌️-------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Catch all episodes, Feature Store, Machine Learning Monitoring and Blogs: https://mlops.community/
Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Ben on LinkedIn: https://www.linkedin.com/in/ben-epstein/
Connect with Kevin on LinkedIn: https://www.linkedin.com/in/kvnkho/
Timestamps:
[00:00] Introduction to Kevin Kho
[04:34] An Intro to Workflow Orchestration with Prefect
[04:51] Kevin's background
[05:16] Agenda
[05:44] Workflow Orchestration
[05:48] Machine Learning Pipeline
[06:14] Pipeline dependencies
[07:06] Points of failures
[08:57] Negative Engineering
[10:17] Introducing Prefect
[12:21] Current state of Prefect
[12:33] Prefect Orion
[13:59] Simple problem
[14:14] Motivation
[12:22] Coindesk
[14:49] Setting a Slack web block explicitly
[20:41] Put infinite loop in production
[22:30] Change Dodgecoin to Bitcoin
[23:58] Testing the detective
[24:22] Function as a flow
[32:13] Intrinsic data passing from task to task
[37:28] Open-source tools
[37:42] Pandera Tutorial
[44:39] Tune Tutorial
[49:45] Evidently.AI Tutorial
[56:16] ML Training
[1:01:40] Resources
[1:02:00] Prefect is hiring!
[1:02:46] MLOps Cool Merch!
[1:03:01] Networking event to match with the MLOps Community