🎟️ the 2025 edition on 27 May: https://aiheroes.it/2025/tickets/
⏩ Chapter:
00:00 Intro
01:51 Why controlling AI?
04:01 What and When to monitor?
09:21 How to monitor a LLM?
12:36 Linear probability
14:21 Perplexity
16:00 Retrieval Confidence Score
17:57 Factual correctness
19:40 AI as a Judge - inconsistency
21:02 Close-ended output
22:29 The Radicalbit AI Monitoring Platform
26:59 The Python SDK
29:53 Conclusions
With the advent of Generative AI, it has become increasingly critical to monitor data and models (of any type) in production. This need has led to the development of numerous techniques and tools dedicated to the field of Monitoring and Observability.
In this talk, I will explore the most effective methods for monitoring AI solutions, covering different applications that include Large Language Models and classical Machine Learning, both in batch and streaming contexts.
I will analyse the common challenges, such as detecting data drift, managing anomalies and controlling model performance over time.
Finally, I will present the open-source AI monitoring solution developed by Radicalbit, a company with extensive experience in MLOps. This solution not only offers the benefits of continuous and in-depth model monitoring, guaranteeing the reliability and efficiency of AI implementations in production, but it also fosters a collaborative environment in which the community continuously improves the solution through contributions.
🎤 MAURO MARINIELLO, Data Scientist
📍AI Heroes is the Italian Artificial Intelligence Conference for Developers, Engineers, Data Scientists & Product Managers.
- The New Workflow to Shape the Future of Business
- AI in the Next Generation Workforce
- Key Technologies, Applications and Trends
ℹ️ inform: https://aiheroes.it/
✅ Tweet: https://twitter.com/AIHeroes_it
✅ Connect: https://www.linkedin.com/showcase/aiheroes-it/
✅ Read: https://t.me/aiheroes
#AIHeroes2025 #AIProduction #AIMonitoring #MLOps #GenerativeAI #OpenSourceAI #DataScience #MachineLearningOps #AIObservability #Radicalbit