Creating custom RAG chunking and embedding strategies with domain specific evaluation experiments
Resources:
Notebook Repo: https://github.com/ALucek/custom-rag-evals/tree/main
Prior Chunking Breakdown: https://youtu.be/Pk2BeaGbcTE?si=pWK6hkXPnJCyb4Ux
ChromaDB Repo: https://github.com/brandonstarxel/chunking_evaluation/tree/main
ChromaDB Colab Notebook: https://colab.research.google.com/drive/1J5ALtDf0_RrswRz2fktjFVeFxe2jbXuJ?usp=sharing#scrollTo=yQiDRQKIf_cl
ChromaDB Research Report: https://research.trychroma.com/evaluating-chunking
Chapters:
00:00 - Why Measure Chunking & Embedding
03:44 - Creating a Custom Chunking Strategy
06:12 - Breaking Down Eval Metrics
08:00 - Metrics: Eval Dataset
09:17 - Metrics: Recall, Precision, IoU
13:27 - General Evals: Describing Test Set
14:32 - General Eval: Process & Running Test
17:58 - General Eval: Embedding Test
19:00 - Running Multiple Evals Across Strategies
20:36 - Multiple Evals: Interpreting Results
23:18 - Domain Specific Dataset Generation & Filtering
27:06 - Running Domain Specific Evals
29:32 - Final Thoughts
#ai #machinelearning #datascience