How RAG Systems Actually Work

Retrieval-augmented generation (RAG) is the most common way to make a language model answer questions from your organisation's own information rather than from whatever it absorbed during training. The idea is simple: find the relevant material first, then ask the model to answer using it. In practice, answer quality depends almost entirely on the unglamorous engineering in between.

Two pipelines, not one

A RAG system is really two pipelines that share an index. The offline pipeline prepares your content so it can be searched. The online pipeline handles each question as it arrives.

Offline:  sources -> ingest -> parse -> chunk -> embed -> index
Online:   question -> retrieve (keyword + vector) -> rerank
          -> assemble prompt -> generate with citations -> log for evaluation

Most of the problems people blame on "the model" actually start in the offline pipeline. If the right information never reaches the index in a usable form, no amount of prompt engineering will get it back.

Preparing the knowledge: from ingestion to indexing

Ingestion connects to the systems where knowledge lives, such as document stores, wikis, ticketing systems, databases and shared drives, and pulls content on a schedule or when something changes. Two details matter more than they first appear. The first is capturing metadata: the source, the author, the last-modified date and the access permissions. The second is handling updates and deletions as well as new documents.

Parsing turns files into clean text while keeping their structure. PDFs, slide decks, spreadsheets and scanned documents each need different handling. Tables, headings, lists and page boundaries carry meaning, and flattening everything into a stream of text throws that meaning away. Scanned documents may need OCR, and complex layouts may need a layout-aware parser.

Chunking splits documents into pieces small enough to retrieve precisely but large enough to make sense on their own. Fixed-size chunks are easy to produce but often cut a sentence or table in half. Structure-aware chunking, which splits on headings, sections or other logical units with some overlap, usually works better. It also helps to attach context to each chunk, such as the document title and section heading, so that a chunk reading "the limit is 30 days" still makes sense when it is retrieved on its own.

Embeddings convert each chunk into a vector that captures its meaning, so that text with similar meaning ends up close together. The embedding model should suit your domain and languages, and it must be the same model you use at query time. If you change it later, you have to re-embed the whole corpus.

Indexing stores the vectors, along with the raw text and metadata, in a vector database or a search engine that supports vectors. A keyword index is usually kept alongside it, for reasons covered next.

Answering a question: from retrieval to generation

Retrieval finds candidate chunks for the user's question. Vector search is good at matching meaning when the wording differs. Keyword search is good at exact terms like product codes, error messages, names, acronyms and clause numbers. Each one misses what the other catches, so production systems commonly use hybrid retrieval: they run both and merge the results. Metadata filters by product, region, date or document type narrow the search further. Some systems also rewrite the query first, for example by expanding an ambiguous follow-up question with context from the conversation.

Reranking takes the top few dozen candidates and reorders them with a more expensive model, usually a cross-encoder that reads the question and each chunk together. It is one of the highest-leverage steps in the pipeline. First-stage retrieval is tuned to find everything that might be relevant, and reranking brings back precision so that only the most relevant handful of chunks reaches the prompt.

Prompt assembly combines the instructions, the selected chunks with their source identifiers, and the user's question. More context is not automatically better, because irrelevant passages dilute the model's attention and add cost and latency. The instructions should tell the model to answer only from the material supplied, and to say so when that material does not contain the answer.

Generation with citations produces the answer and links each claim to the chunks it came from. Citations let users verify the answer and let you audit it, but they only mean something if you check them. A lightweight post-processing step can confirm that each cited source was actually in the context and that the passage plausibly supports the claim.

Permission-aware retrieval

In an enterprise, not everyone is allowed to see every document. A RAG system that puts everything into one pool and filters nothing will happily surface a confidential HR document in answer to an innocent question. Once that text is in the prompt, you cannot count on the model to keep it secret.

The fix is to enforce access control at retrieval time:

1. Capture permissions during ingestion and store them as metadata on each chunk.
2. Filter every query by the identity of the person asking, before results reach the reranker or the prompt.
3. Sync permissions whenever they change, just as you sync content.

For stricter requirements, such as separate customers or regulated data, a physically separate index per tenant is easier to reason about than relying on filters alone.

Why RAG systems fail

Most failure modes happen before the model is involved:

• Bad chunking. Answers get split across chunk boundaries, tables get broken apart, or chunks lose the context needed to interpret them.
• Retrieval misses. The right content exists but is not retrieved. Often the user's vocabulary differs from the document's, or pure vector search misses an exact identifier.
• Stale data. The index lags behind the source, or deleted documents stay in it, so the system answers confidently from an outdated policy.
• Access control leaks. Permissions are not captured or not enforced, so people see content they should not.
• Context overload. Too many loosely relevant chunks crowd out the one that matters.
• Unsupported generation. The model fills gaps with plausible content that is not in the sources, especially when retrieval returned nothing useful.
• Untrusted content in the index. Documents can contain text that reads like instructions. Treat retrieved content as data, never as commands, especially if the system can take actions.

Evaluate retrieval and generation separately

When an answer is wrong, you need to know which half of the system caused it. An end-to-end test only tells you that something failed.

Build a test set of realistic questions, each labelled with the documents or passages that contain the answer. Then measure in two layers:

• Retrieval quality. Did the correct passages appear in the top results, and how high were they ranked? Metrics such as recall at k and mean reciprocal rank answer this directly, without involving the language model at all.
• Generation quality. Given the right context, was the answer correct, complete and faithful to the sources? Were the citations accurate? Did the model decline when the context did not contain the answer?

Measuring the layers separately turns vague complaints into specific fixes. Poor recall points to chunking, embeddings or hybrid search. Good recall with poor answers points to prompt assembly or the model. Run the suite on every change to chunking, embeddings, prompts or models, and add real failures from production as new test cases.

Key takeaways

• RAG quality depends mostly on ingestion, parsing and chunking, not on the model.
• Hybrid retrieval with reranking is a strong, practical default.
• Enforce permissions at retrieval time, using the identity of the person asking.
• Keep the index fresh, including updates and deletions.
• Evaluate retrieval and generation separately so you know what to fix.