Retrieval-Augmented Generation, Explained Without the Hype

Ask a general-purpose language model about your company's internal policy or a document it has never seen, and it will often produce a confident but wrong answer. Retrieval-augmented generation, usually shortened to RAG, is the standard fix. Instead of relying only on what the model learned during training, RAG retrieves relevant text from a knowledge base and gives it to the model as context before it answers.

The core idea

RAG splits the problem into two steps: retrieve, then generate. When a question comes in, the system first searches a collection of documents for the most relevant passages. It then places those passages into the prompt and asks the model to answer using them. The model is no longer guessing from memory; it is summarising text it has been handed.

How retrieval actually works

The retrieval step usually relies on embeddings. An embedding model turns a chunk of text into a vector, a list of numbers that captures its meaning. Chunks with similar meaning end up close together in vector space. When a question arrives, it is embedded the same way, and the system finds the chunks whose vectors are nearest to the question's vector.

The typical pipeline

• Chunking: documents are split into passages small enough to be specific but large enough to keep context.
• Embedding: each chunk is converted to a vector and stored in a vector database.
• Retrieval: the question is embedded and the closest chunks are fetched.
• Generation: the chunks and the question go to the language model, which answers from them.

Why teams choose RAG

RAG has three practical advantages over trying to bake knowledge into a model through fine-tuning. It keeps information current, because you can update the document store at any time without retraining. It is auditable, because answers can cite the passages they came from. And it is cheaper, because indexing documents costs far less than training.

Where it goes wrong

RAG is not magic. If retrieval returns the wrong passages, the model will answer from the wrong context. Poor chunking, a weak embedding model, or a question that does not match how the documents are written can all hurt results. Good RAG systems invest as much in the retrieval quality as in the model itself, and they measure both.

Chunking is quietly the hardest part

The way you split documents has an outsized effect on quality, and it is where most homegrown systems go wrong. Chunks that are too large dilute the relevant sentence among paragraphs of unrelated text, so the model has to work harder to find the answer. Chunks that are too small lose the surrounding context that gives a sentence its meaning. The reliable approach is to split along natural boundaries such as headings, paragraphs or sections, and to let chunks overlap slightly so a fact that straddles a boundary is not cut in half.

Better retrieval than pure vectors

Embeddings capture meaning well but can miss exact terms, such as a product code or an error number that has to match literally. The current best practice is hybrid retrieval: combine vector similarity with traditional keyword search so you get both semantic matches and exact ones. Many systems then add a re-ranking step, where a second, more precise model reorders the initial candidates so the strongest passages land at the top of the prompt, where the model pays them the most attention.

How to know it is working

Because RAG has two stages, you should measure two things. Retrieval quality asks whether the right passages were fetched at all; if the answer is not in the retrieved context, no model can produce it. Generation quality asks whether the model used that context faithfully instead of drifting back to its own memory. Keeping a small set of real questions with known good answers, and checking both stages against it whenever you change the pipeline, is what separates a RAG system that improves over time from one that quietly regresses.

The bottom line

If you want a model to answer from your own documents with citations you can check, RAG is the pragmatic approach. It sidesteps the cost and staleness of fine-tuning and turns a general model into something that speaks to your data.