Optimize the Embedding Space for Improving RAG

Embeddings are vector representations that capture the semantic meaning of words or sentences. Besides having quality data, selecting a very good embedding model is a very powerful and underrated step for optimizing your RAG application. Multilingual models are especially difficult as most are pre-trained on English data. The fitting embeddings make an enormous difference — don’t just grab the primary model you see!

The semantic space determines the relationships between words and ideas. An accurate semantic space improves retrieval performance. Inaccurate embeddings result in irrelevant chunks or missing information. A greater model directly improves your RAG system’s capabilities.

In this text, we are going to create a question-answer dataset from PDF documents to be able to find the very best model for our task and language. During RAG, if the expected answer is retrieved, it means the embedding model positioned the query and answer close enough within the semantic space.

While we deal with French and Italian, the method may be adapted to any language because the very best embeddings might differ.

Embedding Models

There are two fundamental sorts of embedding models: static and dynamic. Static embeddings like word2vec generate a vector for every word. The vectors are combined, often by averaging, to create a final embedding. These kinds of embeddings are usually not often utilized in production anymore because they don’t consider how a word’s meaning can change in function to the encircling words.

Dynamic embeddings are based on Transformers like BERT, which incorporate context awareness through self-attention layers, allowing them to represent words based on the encircling context.

Most current fine-tuned models use contrastive learning. The model learns semantic similarity by seeing each positive and negative text pairs during training.