Large Language Models (LLMs) have a data freshness problem. Even some of the most powerful models, like ChatGPT's gpt-3.5-turbo and GPT-4, have no idea about recent events.
The world, according to LLMs, is frozen in time. They only know the world as it appeared through their training data.
So, how do we handle this problem? We can use retrieval augmentation. This technique allows us to retrieve relevant information from an external knowledge base and give that information to our LLM.
The external knowledge base is our "window" into the world beyond the LLM's training data. In this video, we will learn all about implementing retrieval augmentation for LLMs using LangChain and the Pinecone vector database.
? Pinecone article:
https://pinecone.io/learn/langchain-retrieval-augmentation
? Code notebook:
https://github.com/pinecone-io/examples/blob/master/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb
?? NLP + LLM Consulting:
https://aurelio.ai
? Discord:
https://discord.gg/c5QtDB9RAP
?️ Support me on Patreon:
https://patreon.com/JamesBriggs
? 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5
? Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership
00:00 Hallucination in LLMs
01:32 Types of LLM Knowledge
03:08 Data Preprocessing with LangChain
09:54 Creating Embeddings with OpenAI's Ada 002
13:14 Creating the Pinecone Vector Database
16:57 Indexing Data into Our Database
20:27 Querying with LangChain
23:07 Generative Question-Answering with LangChain
25:27 Adding Citations to Generated Answers
28:42 Summary of Retrieval Augmentation in LangChain
#artificialintelligence #langchain #nlp #openai #deeplearning #chatgpt