Selecting the model that works best on your data
OpenAI recently released their latest generation of embedding models, called embedding v3, which they describe as their most performant embedding models, with higher multilingual performances. The models are available in two classes: a smaller one called text-embedding-3-small, and a bigger and more powerful one called text-embedding-3-large.
Little or no information was disclosed regarding the way these models were designed and trained. As their previous embedding model release (December 2022 with the ada-002 model class), OpenAI again chooses a closed-source approach where the models may only be accessed through a paid API.
But are the performances so good that they make it price paying?
The motivation for this post is to empirically compare the performances of those latest models with their open-source counterparts. We’ll depend on an information retrieval workflow, where essentially the most relevant documents in a corpus need to be found given a user query.
Our corpus will likely be the European AI Act, which is currently in its final stages of validation. An interesting characteristic of this corpus, besides being the first-ever legal framework on AI worldwide, is its availability in 24 languages. This makes it possible to check the accuracy of knowledge retrieval across different families of languages.
The post will undergo the 2 principal following steps:
- Generate a custom synthetic query/answer dataset from a multilingual text corpus
- Compare the accuracy of OpenAI and state-of-the-art open-source embedding models on this tradition dataset.
The code and data to breed the outcomes presented on this post are made available on this Github repository. Note that the EU AI Act is used for example, and the methodology followed on this post may be adapted to other data corpus.
Allow us to first start by generating a dataset of questions and answers (Q/A) on custom data, which will likely be used to evaluate the performance of various embedding models. The advantages of generating a custom Q/A dataset are twofold. First, it avoids biases by ensuring that the dataset has not been a part of the training of an embedding model, which can occur on reference benchmarks akin to MTEB. Second, it allows to tailor the assessment to a particular corpus of knowledge, which may be relevant within the case of retrieval augmented applications (RAG) for instance.
We’ll follow the easy process suggested by Llama Index of their documentation. The corpus is first split right into a set of chunks. Then, for every chunk, a set of synthetic questions are generated by the use of a big language model (LLM), such that the reply lies within the corresponding chunk. The method is illustrated below:
Implementing this strategy is easy with an information framework for LLM akin to Llama Index. The loading of the corpus and splitting of text may be conveniently carried out using high-level functions, as illustrated with the next code.
from llama_index.readers.web import SimpleWebPageReader
from llama_index.core.node_parser import SentenceSplitterlanguage = "EN"
url_doc = "https://eur-lex.europa.eu/legal-content/"+language+"/TXT/HTML/?uri=CELEX:52021PC0206"
documents = SimpleWebPageReader(html_to_text=True).load_data([url_doc])
parser = SentenceSplitter(chunk_size=1000)
nodes = parser.get_nodes_from_documents(documents, show_progress=True)
In this instance, the corpus is the EU AI Act in English, taken directly from the Web using this official URL. We use the draft version from April 2021, as the ultimate version just isn’t yet available for all European languages. On this version, English language may be replaced within the URL by any of the 23 other EU official languages to retrieve the text in a unique language (BG for Bulgarian, ES for Spanish, CS for Czech, and so forth).
We use the SentenceSplitter object to separate the document in chunks of 1000 tokens. For English, this ends in about 100 chunks.
Each chunk is then provided as context to the next prompt (the default prompt suggested within the Llama Index library):
prompts={}
prompts["EN"] = """
Context information is below.---------------------
{context_str}
---------------------
Given the context information and never prior knowledge, generate only questions based on the below query.
You might be a Teacher/ Professor. Your task is to setup {num_questions_per_chunk} questions for an upcoming quiz/examination.
The questions ought to be diverse in nature across the document. Restrict the inquiries to the context information provided."
"""
The prompt goals at generating questions on the document chunk, as if a teacher were preparing an upcoming quiz. The variety of inquiries to generate for every chunk is passed because the parameter ‘num_questions_per_chunk’, which we set to 2. Questions can then be generated by calling the generate_qa_embedding_pairs from the Llama Index library:
from llama_index.llms import OpenAI
from llama_index.legacy.finetuning import generate_qa_embedding_pairsqa_dataset = generate_qa_embedding_pairs(
llm=OpenAI(model="gpt-3.5-turbo-0125",additional_kwargs={'seed':42}),
nodes=nodes,
qa_generate_prompt_tmpl = prompts[language],
num_questions_per_chunk=2
)
We rely for this task on the GPT-3.5-turbo-0125 mode from OpenAI, which is in line with OpenAI the flagship model of this family, supporting a 16K context window and optimized for dialog (https://platform.openai.com/docs/models/gpt-3-5-turbo).
The resulting objet ‘qa_dataset’ accommodates the questions and answers (chunks) pairs. For instance of generated questions, here is the result for the primary two questions (for which the ‘answer’ is the primary chunk of text):
1) What are the principal objectives of the proposal for a Regulation laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) in line with the explanatory memorandum?
2) How does the proposal for a Regulation on artificial intelligence aim to deal with the risks related to the usage of AI while promoting the uptake of AI within the European Union, as outlined within the context information?
The variety of chunks and questions is dependent upon the language, starting from around 100 chunks and 200 questions for English, to 200 chunks and 400 questions for Hungarian.
Our evaluation function follows the Llama Index documentation and consists in two principal steps. First, the embeddings for all answers (document chunks) are stored in a VectorStoreIndex for efficient retrieval. Then, the evaluation function loops over all queries, retrieves the highest k most similar documents, and the accuracy of the retrieval in assessed by way of MRR (Mean Reciprocal Rank).
def evaluate(dataset, embed_model, insert_batch_size=1000, top_k=5):
# Get corpus, queries, and relevant documents from the qa_dataset object
corpus = dataset.corpus
queries = dataset.queries
relevant_docs = dataset.relevant_docs# Create TextNode objects for every document within the corpus and create a VectorStoreIndex to efficiently store and retrieve embeddings
nodes = [TextNode(id_=id_, text=text) for id_, text in corpus.items()]
index = VectorStoreIndex(
nodes, embed_model=embed_model, insert_batch_size=insert_batch_size
)
retriever = index.as_retriever(similarity_top_k=top_k)
# Prepare to gather evaluation results
eval_results = []
# Iterate over each query within the dataset to judge retrieval performance
for query_id, query in tqdm(queries.items()):
# Retrieve the top_k most similar documents for the present query and extract the IDs of the retrieved documents
retrieved_nodes = retriever.retrieve(query)
retrieved_ids = [node.node.node_id for node in retrieved_nodes]
# Check if the expected document was among the many retrieved documents
expected_id = relevant_docs[query_id][0]
is_hit = expected_id in retrieved_ids # assume 1 relevant doc per query
# Calculate the Mean Reciprocal Rank (MRR) and append to results
if is_hit:
rank = retrieved_ids.index(expected_id) + 1
mrr = 1 / rank
else:
mrr = 0
eval_results.append(mrr)
# Return the common MRR across all queries as the ultimate evaluation metric
return np.average(eval_results)
The embedding model is passed to the evaluation function by the use of the `embed_model` argument, which for OpenAI models is an OpenAIEmbedding object initialised with the name of the model, and the model dimension.
from llama_index.embeddings.openai import OpenAIEmbeddingembed_model = OpenAIEmbedding(model=model_spec['model_name'],
dimensions=model_spec['dimensions'])
The dimensions API parameter can shorten embeddings (i.e. remove some numbers from the top of the sequence) without the embedding losing its concept-representing properties. OpenAI for instance suggests of their annoucement that on the MTEB benchmark, an embedding may be shortened to a size of 256 while still outperforming an unshortened text-embedding-ada-002 embedding with a size of 1536.
We ran the evaluation function on 4 different OpenAI embedding models:
- two versions of
text-embedding-3-large: one with the bottom possible dimension (256), and the opposite one with the very best possible dimension (3072). These are called ‘OAI-large-256’ and ‘OAI-large-3072’. - OAI-small: The
text-embedding-3-smallembedding model, with a dimension of 1536. - OAI-ada-002: The legacy
text-embedding-ada-002model, with a dimension of 1536.
Each model was evaluated on 4 different languages: English (EN), French (FR), Czech (CS) and Hungarian (HU), covering examples of Germanic, Romance, Slavic and Uralic language, respectively.
embeddings_model_spec = {
}embeddings_model_spec['OAI-Large-256']={'model_name':'text-embedding-3-large','dimensions':256}
embeddings_model_spec['OAI-Large-3072']={'model_name':'text-embedding-3-large','dimensions':3072}
embeddings_model_spec['OAI-Small']={'model_name':'text-embedding-3-small','dimensions':1536}
embeddings_model_spec['OAI-ada-002']={'model_name':'text-embedding-ada-002','dimensions':None}
results = []
languages = ["EN", "FR", "CS", "HU"]
# Loop through all languages
for language in languages:
# Load dataset
file_name=language+"_dataset.json"
qa_dataset = EmbeddingQAFinetuneDataset.from_json(file_name)
# Loop through all models
for model_name, model_spec in embeddings_model_spec.items():
# Get model
embed_model = OpenAIEmbedding(model=model_spec['model_name'],
dimensions=model_spec['dimensions'])
# Assess embedding rating (by way of MRR)
rating = evaluate(qa_dataset, embed_model)
results.append([language, model_name, score])
df_results = pd.DataFrame(results, columns = ["Language" ,"Embedding model", "MRR"])
The resulting accuracy by way of MRR is reported below:
As expected, for the massive model, higher performances are observed with the larger embedding size of 3072. Compared with the small and legacy Ada models, the massive model is nonetheless smaller than we might have expected. For comparison, we also report below the performances obtained by the OpenAI models on the MTEB benchmark.
It’s interesting to notice that the differences in performances between the massive, small and Ada models are much less pronounced in our assessment than within the MTEB benchmark, reflecting the incontrovertible fact that the common performances observed in large benchmarks don’t necessarily reflect those obtained on custom datasets.
The open-source research around embeddings is sort of energetic, and latest models are commonly published. A superb place to maintain updated in regards to the latest published models is the Hugging Face 😊 MTEB leaderboard.
For the comparison in this text, we chosen a set of 4 embedding models recently published (2024). The factors for selection were their average rating on the MTEB leaderboard and their ability to take care of multilingual data. A summary of the principal characteristics of the chosen models are reported below.
- E5-Mistral-7B-instruct (E5-mistral-7b): This E5 embedding model by Microsoft is initialized from Mistral-7B-v0.1 and fine-tuned on a combination of multilingual datasets. The model performs best on the MTEB leaderboard, but can be by far the most important one (14GB).
- multilingual-e5-large-instruct (ML-E5-large): One other E5 model from Microsoft, meant to raised handle multilingual data. It’s initialized from xlm-roberta-large and trained on a combination of multilingual datasets. It is far smaller (10 times) than E5-Mistral, but in addition has a much lower context size (514).
- BGE-M3: The model was designed by the Beijing Academy of Artificial Intelligence, and is their state-of-the-art embedding model for multilingual data, supporting greater than 100 working languages. It was not yet benchmarked on the MTEB leaderboard as of twenty-two/02/2024.
- nomic-embed-text-v1 (Nomic-Embed): The model was designed by Nomic, and claims higher performances than OpenAI Ada-002 and text-embedding-3-small while being only 0.55GB in size. Interestingly, the model is the primary to be fully reproducible and auditable (open data and open-source training code).
The code for evaluating these open-source models is analogous to the code used for OpenAI models. The principal change lies within the model specifications, where additional details akin to maximum context length and pooling types need to be specified. We then evaluate each model for every of the 4 languages:
embeddings_model_spec = {
}embeddings_model_spec['E5-mistral-7b']={'model_name':'intfloat/e5-mistral-7b-instruct','max_length':32768, 'pooling_type':'last_token',
'normalize': True, 'batch_size':1, 'kwargs': {'load_in_4bit':True, 'bnb_4bit_compute_dtype':torch.float16}}
embeddings_model_spec['ML-E5-large']={'model_name':'intfloat/multilingual-e5-large','max_length':512, 'pooling_type':'mean',
'normalize': True, 'batch_size':1, 'kwargs': {'device_map': 'cuda', 'torch_dtype':torch.float16}}
embeddings_model_spec['BGE-M3']={'model_name':'BAAI/bge-m3','max_length':8192, 'pooling_type':'cls',
'normalize': True, 'batch_size':1, 'kwargs': {'device_map': 'cuda', 'torch_dtype':torch.float16}}
embeddings_model_spec['Nomic-Embed']={'model_name':'nomic-ai/nomic-embed-text-v1','max_length':8192, 'pooling_type':'mean',
'normalize': True, 'batch_size':1, 'kwargs': {'device_map': 'cuda', 'trust_remote_code' : True}}
results = []
languages = ["EN", "FR", "CS", "HU"]
# Loop through all models
for model_name, model_spec in embeddings_model_spec.items():
print("Processing model : "+str(model_spec))
# Get model
tokenizer = AutoTokenizer.from_pretrained(model_spec['model_name'])
embed_model = AutoModel.from_pretrained(model_spec['model_name'], **model_spec['kwargs'])
if model_name=="Nomic-Embed":
embed_model.to('cuda')
# Loop through all languages
for language in languages:
# Load dataset
file_name=language+"_dataset.json"
qa_dataset = EmbeddingQAFinetuneDataset.from_json(file_name)
start_time_assessment=time.time()
# Assess embedding rating (by way of hit rate at k=5)
rating = evaluate(qa_dataset, tokenizer, embed_model, model_spec['normalize'], model_spec['max_length'], model_spec['pooling_type'])
# Get duration of rating assessment
duration_assessment = time.time()-start_time_assessment
results.append([language, model_name, score, duration_assessment])
df_results = pd.DataFrame(results, columns = ["Language" ,"Embedding model", "MRR", "Duration"])
The resulting accuracies by way of MRR are reported below.
BGE-M3 seems to supply the very best performances, followed on average by ML-E5-Large, E5-mistral-7b and Nomic-Embed. BGE-M3 model just isn’t yet benchmarked on the MTEB leaderboard, and our results indicate that it could rank higher than other models. Additionally it is interesting to notice that while BGE-M3 is optimized for multilingual data, it also performs higher for English than the opposite models.
We moreover report the processing times for every embedding model below.
The E5-mistral-7b, which is greater than 10 times larger than the opposite models, is without surprise by far the slowest model.
Allow us to put side-by-side of the performance of the eight tested models in a single figure.
The important thing observations from these results are:
- Best performances were obtained by open-source models. The BGE-M3 model, developed by the Beijing Academy of Artificial Intelligence, emerged as the highest performer. The model has the identical context length as OpenAI models (8K), for a size of two.2GB.
- Consistency Across OpenAI’s Range. The performances of the massive (3072), small and legacy OpenAI models were very similar. Reducing the embedding size of the massive model (256) nonetheless led to a degradation of performances.
- Language Sensitivity. Just about all models (except ML-E5-large) performed best on English. Significant variations in performances were observed in languages like Czech and Hungarian.
Must you due to this fact go for a paid OpenAI subscription, or for hosting an open-source embedding model?
OpenAI’s recent price revision has made access to their API significantly cheaper, with the price now standing at $0.13 per million tokens. Coping with a million queries per thirty days (and assuming that every query involves around 1K token) would due to this fact cost on the order of $130. Depending in your use case, it could due to this fact not be cost-effective to rent and maintain your individual embedding server.
Cost-effectiveness is nonetheless not the only consideration. Other aspects akin to latency, privacy, and control over data processing workflows can also should be considered. Open-source models offer the advantage of complete data control, enhancing privacy and customization. Alternatively, latency issues have been observed with OpenAI’s API, sometimes leading to prolonged response times.
In conclusion, the selection between open-source models and proprietary solutions like OpenAI’s doesn’t lend itself to an easy answer. Open-source embeddings present a compelling option, combining performance with greater control over data. Conversely, OpenAI’s offerings should still appeal to those prioritizing convenience, especially if privacy concerns are secondary.
Notes: