Ramesh Johari (Stanford, Uber, Airbnb, and more) explores the art and science of online experimentation, especially in the context of marketplaces and tech companies.
Ramesh shares insights on how organizations evolve from basic experimentation practices to becoming fast, adaptive, and self learning organizations. We dive into challenges like the risk aversion trap, the importance of learning from negative results, and how generative AI is reshaping the experimentation landscape.
We also talk about common failure modes and the types of things you're probably doing wrong, along with strategies to avoid these pitfalls. Plus, we discussed the role of incentives, the necessity of data driven decision making, and what it means to experiment in high stakes environments.
You can find more on our website: https://high-signal.delphina.ai/
0:00 Introduction
1:00 Episode Overview
3:00 The Future of Experimentation
6:00 Meet the Team at Delphina
7:30 Ramesh’s Background
9:30 What is Experimentation?
12:00 Evolution of Experimentation in Organizations
16:00 The Risk Aversion Cycle
18:00 Incentives in Experimentation
20:00 Fat Tails in Experimentation
24:00 The Role of Dogfooding
26:00 Experimentation and Prediction
29:00 Encouraging a Culture of Experimentation
32:00 Embedding Data Scientists
34:00 Generative AI and Experimentation
38:00 AI’s Role in Managing Experimentation Data
40:00 The Self-Learning Organization
42:00 Limitations of Experiments
46:00 Experimentation vs. Innovation
49:00 Closing Thoughts and Contact Info
50:30 Outro