Back to Study Material Deep Learning

Retrieval augmented generation

Retrieval augmented generation, commonly abbreviated as RAG, is a technique that combines a language model with an external retrieval system. Instead of as...

Retrieval augmented generation

The Core Idea Behind RAG

Retrieval augmented generation, commonly abbreviated as RAG, is a technique that combines a language model with an external retrieval system. Instead of asking the model to answer a question from its parametric memory alone, RAG first retrieves relevant documents from an external knowledge base and then provides them as context in the prompt alongside the question. The model generates its answer by synthesizing the retrieved documents rather than relying entirely on knowledge baked into its weights during training.

A customer support application illustrates the benefit. A language model trained on general internet data will not know the specific return policies of a particular retailer. A RAG system stores the retailer's policy documents in a searchable index, retrieves the most relevant passages when a customer asks a policy-related question, and injects those passages into the prompt so the model can answer accurately and specifically.

Exam tip: RAG addresses two key limitations of parametric language models: knowledge staleness and hallucination. Since retrieved documents can be updated without retraining, the system's knowledge remains current. Grounding the generation in explicit retrieved text reduces the tendency to fabricate facts, though it does not eliminate hallucination entirely.

Retrieval Methods and Chunking Strategies

The retrieval component of a RAG system can use sparse retrieval, dense retrieval, or a combination. Sparse retrieval, implemented with systems like BM25, scores documents based on keyword overlap with the query and is fast and interpretable. Dense retrieval uses embedding models to encode both the query and documents into vector representations and retrieves by similarity in that vector space, capturing semantic rather than just lexical relevance.

Document chunking, the process of splitting long documents into segments for indexing, significantly affects retrieval quality. Chunks that are too small may lack sufficient context to be useful on their own. Chunks that are too large may dilute the relevance signal and exceed the model's context window. Typical chunk sizes range from 256 to 1024 tokens, with overlap between adjacent chunks to ensure that content spanning a chunk boundary is not lost.

Hybrid retrieval systems that combine BM25 scores with dense retrieval scores often outperform either approach alone, because the two methods have complementary failure modes and their combination covers a broader range of query types.

Re-ranking, Context Injection, and Evaluation

After initial retrieval, a re-ranking step can improve the relevance of the final context. A cross-encoder re-ranker jointly encodes the query and each candidate passage and scores their relevance more accurately than a bi-encoder, which encodes them independently. Because cross-encoders are slower, they are typically applied to a small set of top candidates rather than the full index.

Context injection is the process of formatting retrieved passages into the language model's prompt. The order and format of injected passages affects generation quality. Research has found that language models tend to pay more attention to information at the beginning and end of the context than in the middle, a phenomenon called the lost-in-the-middle effect. Placing the most relevant passages at the start of the context is a practical mitigation.

Evaluating a RAG system requires measuring both retrieval quality and end-to-end generation quality separately. Retrieval can be evaluated with recall at k metrics. Generation can be evaluated with faithfulness, measuring whether the answer is grounded in the retrieved context, and answer relevance, measuring whether it addresses the question. Combined metrics like RAGAS provide a structured framework for this evaluation.