Find out how to use UMAP dimensionality reduction for Embeddings to indicate multiple evaluation Questions and their relationships to source documents with Ragas, OpenAI, Langchain and ChromaDB
18 hours ago
Retrieval-Augmented Generation (RAG) adds a retrieval step to the workflow of an LLM, enabling it to question relevant data from additional sources like private documents when responding to questions and queries [1]. This workflow doesn’t require costly training or fine-tuning of LLMs on the extra documents. The documents are split into snippets, that are then indexed, often using a compact ML-generated vector representation (embedding). Snippets with similar content can be in proximity to one another on this embedding space.
The RAG application projects the user-provided questions into the embedding space to retrieve relevant document snippets based on their distance to the query. The LLM can use the retrieved information to reply the query and to substantiate its conclusion by presenting the snippets as references.
The evaluation of a RAG application is difficult [2]. Different approaches exist: on one hand, there are methods where the reply as ground truth have to be provided by the developer; then again, the reply (and the query) can be generated by one other LLM. One among the most important open-source systems for LLM-supported answering is Ragas [4](Retrieval-Augmented Generation Assessment), which provides
- Methods for generating test data based on the documents and
- Evaluations based on different metrics for evaluating retrieval and generation steps one-by-one and end-to-end.
In this text, you’ll learn
The code is out there at Github
Start a notebook and install the required python packages
!pip install langchain langchain-openai chromadb renumics-spotlight
%env OPENAI_API_KEY=
This tutorial uses the next python packages:
- Langchain: A framework to integrate language models and RAG components, making the setup process smoother.
- Renumics-Highlight: A visualization tool to interactively explore unstructured ML datasets.
- Ragas: a framework that helps you evaluate your RAG pipelines
Disclaimer: The writer of this text can also be certainly one of the developers of Highlight.
You need to use your personal RAG Application, skip to the subsequent part to learn the best way to evaluate, extract and visualize.
Or you should use the RAG application from the last article with our prepared dataset of all Formula One articles of Wikipedia. There you can too insert your personal Documents right into a ‘docs/’ subfolder.
This dataset relies on articles from Wikipedia and is licensed under the Creative Commons Attribution-ShareAlike License. The unique articles and a listing of authors may be found on the respective Wikipedia pages.
Now you should use Langchain’s DirectoryLoader to load all files from the docs subdirectory and split the documents in snippets using the RecursiveCharacterTextSpliter. With OpenAIEmbeddings you’ll be able to create embeddings and store them in a ChromaDB as vector store. For the Chain itself you should use LangChains ChatOpenAI and a ChatPromptTemplate.
The linked code for this text comprises all needed steps and you could find an in depth description of all steps above within the last article.
One vital point is, that it’s best to use a hash function to create ids for snippets in ChromaDB. This permits to search out the embeddings within the db should you only have the document with its content and metadata. This makes it possible to skip documents that exist already within the database.
import hashlib
import json
from langchain_core.documents import Documentdef stable_hash_meta(doc: Document) -> str:
"""
Stable hash document based on its metadata.
"""
return hashlib.sha1(json.dumps(doc.metadata, sort_keys=True).encode()).hexdigest()
...
splits = text_splitter.split_documents(docs)
splits_ids = [
{"doc": split, "id": stable_hash_meta(split.metadata)} for split in splits
]
existing_ids = docs_vectorstore.get()["ids"]
new_splits_ids = [split for split in splits_ids if split["id"] not in existing_ids]
docs_vectorstore.add_documents(
documents=[split["doc"] for split in new_splits_ids],
ids=[split["id"] for split in new_splits_ids],
)
docs_vectorstore.persist()
For a typical topic like Formula One, one also can use ChatGPT on to generate general questions. In this text, 4 methods of query generation are used:
- GPT4: 30 questions were generated using ChatGPT 4 with the next prompt “Write 30 query about Formula one”
– Random Example: “Which Formula 1 team is thought for its prancing horse logo?” - GPT3.5: One other 199 query were generated with ChatGPT 3.5 with the next prompt “Write 100 query about Formula one” and repeating “Thanks, write one other 100 please”
– Example: “”Which driver won the inaugural Formula One World Championship in 1950?” - Ragas_GPT4: 113 questions were generated using Ragas. Ragas utilizes the documents again and its own embedding model to construct a vector database, which is then used to generate questions with GPT4.
– Example: “Are you able to tell me more concerning the performance of the Jordan 198 Formula One automotive within the 1998 World Championship?” - Rags_GPT3.5: 226 additional questions were generated with Ragas — here we use GPT3.5
– Example: “What incident occurred on the 2014 Belgian Grand Prix that led to Hamilton’s retirement from the race?”
from ragas.testset import TestsetGeneratorgenerator = TestsetGenerator.from_default(
openai_generator_llm="gpt-3.5-turbo-16k",
openai_filter_llm="gpt-3.5-turbo-16k"
)
testset_ragas_gpt35 = generator.generate(docs, 100)
The questions and answers weren’t reviewed or modified in any way. All questions are combined in a single dataframe with the columns id, query, ground_truth, question_by and answer.
Next, the questions can be posed to the RAG system. For over 500 questions, this will take a while and incur costs. In case you ask the questions row-by-row, you’ll be able to pause and proceed the method or recuperate from a crash without losing the outcomes to this point:
for i, row in df_questions_answers.iterrows():
if row["answer"] is None or pd.isnull(row["answer"]):
response = rag_chain.invoke(row["question"])df_questions_answers.loc[df_questions_answers.index[i], "answer"] = response[
"answer"
]
df_questions_answers.loc[df_questions_answers.index[i], "source_documents"] = [
stable_hash_meta(source_document.metadata)
for source_document in response["source_documents"]
]
Not only is the reply stored but additionally the source IDs of the retrieved document snippets, and their text content as context: