Reproducible Machine Learning How Not to Repeat the Past

Speaker's Bio: Stacey Svetlichnaya Deep Learning Engineer, Weights & Biases Stacey Svetlichnaya is a deep learning engineer at Weights & Biases, building developer tools for visualization, explainability, reproducibility, and collaboration in AI. She’s not sure if the climate crisis or AI safety is the bigger existential threat, so she strives to maximize impact on both. She enjoys the intersection of machine learning research, application, and UX, mostly for vision & language models (image aesthetic quality and style classification, object recognition, caption generation, and emoji semantics). Previously, she worked on image search, productionizing ML systems, and discovery & recommendation on Flickr, following the acquisition of LookFlow, a visual similarity search engine. Stacey holds a Stanford BS ‘11 and MS ’12 in Symbolic Systems, focusing on neuroscience. Abstract: Reliably reproducing deep learning research—whether from others' code or your own past attempts—is a notorious challenge in machine learning. What are some useful patterns for keeping your experiments organized and repeatable? Standardized logging, granular and annotated versions of datasets and model checkpoints, and templated workflows for analysis and visualization can help capture the full scope of an ML project. We will cover general practices for building reproducible training and inference pipelines, finding and recreating individual steps or checkpoints, and presenting your analysis for more effective collaboration. The easier it becomes to revisit earlier states of your whole team's development cycle, the more confidently you can train models, minimizing the need to duplicate effort or redo important steps.