In this special live-streamed recording of Vanishing Gradients, Hugo speaks with Eugene Yan (Amazon), Bryan Bischof (Hex), Charles Frye (Modal), Hamel Husain (Parlance Labs), Jason Liu (Instructor), and Shreya Shankar (UC Berkeley).
Over the past year, these six guests have been building real-world applications on top of LLMs. They have identified crucial and often neglected lessons that are essential for developing and building AI products.
They have recently written an O’Reilly report based on these learnings and, in this conversation, they share advice and lessons for anyone who wants to build products informed by LLMs, ranging from tactical to operational and strategic.
You can find the podcast episodes here or on your app of choice:
https://vanishinggradients.fireside.fm/29
https://vanishinggradients.fireside.fm/30
They cover
* Prompting and information retrieval/RAG,
* Tuning and optimizing workflows,
* Evaluation and monitoring,
* Data and working with models,
* Product, teams, and roles,
* MVPs and the ideas behind “No GPUs before PMF”,
* Key aspects of quick iteration,
* How and why to start with prompting, evals, and data collection,
And much more!
0:03 - Hugo introduces the livestream and guests
3:28 - Eugene's background and interest in LLMs
4:32 - Shreya's background and interest in LLMs
6:35 - Hamel's background and experience with LLMs
9:37 - Origin story of the report
11:32 - Discussion on evaluating LLM output quality
18:17 - Importance of evaluations (evals) in LLM development
25:49 - Discussion on data literacy and looking at data
30:33 - Debate on AI engineer role and necessary skills
54:29 - Discussion on fine-tuning vs. off-the-shelf models
57:48 - Bryan and Charles join the conversation
1:24:29 - Debate on LLMOps tools and processes
1:37:37 - Discussion on composability in LLM systems
1:50:33 - Excitement about future developments in AI/LLMs
2:20:32 - Strategies for building trust with stakeholders
2:24:56 - Importance of systems-level view in LLM applications
2:31:50 - Key skills for working with LLMs
2:45:12 - Final advice for those starting with LLM development