RAG
Retrieval-augmented generation fetches relevant passages from a controlled knowledge base at answer time instead of trusting model memory.
RAG retrieves relevant passages from a knowledge base and gives them to the model at answer time. It keeps facts in a controlled source instead of relying on model memory.
RAG retrieves relevant passages from a knowledge base and gives them to the model at answer time. It keeps facts in a controlled source instead of relying on model memory.
All glossary terms
The short version
- Retrieval-augmented generation retrieves passages from a source you control and places them in the model’s context at answer time, so the answer is grounded in text you can point to rather than in model memory.
- The quality ceiling is set by retrieval. If the right passage is not in the top handful of results, no prompt, model or temperature setting will recover it.
- Chunking is the most consequential decision and the least discussed. Chunks must be small enough to be specific and large enough to be self-contained, with the heading carried into each one.
- Reranking a wider candidate set is usually the cheapest quality gain available, and hybrid retrieval beats pure vector search on part numbers, names and Hinglish queries.
- RAG does not fix bad source content. It retrieves it faithfully, cites it confidently and makes the error reproducible, which is worse than a model that admits ignorance.
- Measure recall at k, faithfulness to the retrieved passage and citation precision. Answer quality without retrieval quality is a coin flip you cannot debug.
What retrieval-augmented generation is
Retrieval-augmented generation is a pattern, not a model. Before the model answers, a retrieval step searches an index of your documents for passages relevant to the question, and those passages are inserted into the prompt as context. The model is then asked to answer using that context. The effect is that facts live in a source you can update, inspect and cite, rather than in parameters you cannot edit and cannot audit.
The pattern exists because model memory is bad at three things. It goes stale, because training has a cutoff. It is unspecific, because the model has read the public internet and not your price list. And it is unfalsifiable, because when the model is wrong there is no passage to inspect. RAG addresses all three by moving the factual claim out of the weights and into a document that a human can check in ten seconds.
The mechanics are a loop with five stages: ingest and chunk the source, embed the chunks into vectors, retrieve candidates for a query, rerank them, and generate an answer with citations. Each stage has its own failure modes, and a defect in any one of them surfaces as the same symptom, which is a confident wrong answer. That is why teams that tune only the prompt rarely fix anything.
What RAG is not
It is not fine-tuning. Fine-tuning changes how a model behaves, its tone, format and task competence; it is a poor way to install facts because the facts are compressed into weights and cannot be updated or cited. RAG changes what the model can see at answer time. Most teams that think they need fine-tuning need better retrieval, and most teams that fine-tune to add knowledge end up doing RAG afterwards anyway.
It is not long context. A million-token window lets you paste more, and pasting everything is not retrieval: it is expensive, it dilutes attention, and it fails exactly when the corpus grows past the window. Long context is useful for a single large document you must reason over in full. It is not a substitute for an index over ten thousand documents, and the cost difference at scale is severe.
Nor is it a fact-checker, a permissions system or a guarantee. RAG retrieves what is in the index, including the outdated page nobody deleted and the internal note that should never have been indexed. It does not know which of two contradictory passages is current unless you tell it. And it does not enforce who may see what unless the retrieval step filters by the caller’s identity before the model sees anything.
Chunking decides what retrieval can find
A chunk is the unit of retrieval, so its boundaries determine what can ever be found. Split a policy document every five hundred characters and you will cut a refund rule in half, retrieve the half that says exceptions apply, and generate an answer that is confidently incomplete. Split it by page and you will retrieve a page of boilerplate with the answer buried in paragraph nine, which the reranker may or may not rescue.
The structure-aware approach is to chunk along the document’s own boundaries: headings, sections, list items, table rows. Attach the heading path to every chunk so that a fragment carries its context, and prepend a one-line summary of the parent section. A chunk that begins with Refunds, international orders, exceptions is dramatically more retrievable than the same sentences with no label. This single change often does more than swapping the embedding model.
Size is a trade. Somewhere between two hundred and five hundred tokens with a small overlap is a reasonable starting range for prose, smaller for FAQs and larger for narrative that loses meaning when cut. Tables need their header row repeated in every chunk, or the numbers become uninterpretable. Code needs function boundaries. There is no universal number, and any vendor who gives you one without asking about your documents is guessing.
Embeddings measure similarity, not truth
An embedding model maps text to a vector so that texts with similar meaning sit close together. Retrieval finds the nearest vectors to the query vector. That is the whole mechanism, and it explains the failure modes. Similarity is about meaning, not about being correct, current or authoritative. An outdated policy and its replacement are near-identical vectors, so both come back, and the model may quote the one that ranks marginally higher.
The choice of embedding model matters less than teams expect, provided it is decent and matched to the language mix. It matters enormously that the query and the documents are embedded by the same model, at the same version, with the same normalisation. Re-embedding half a corpus after an upgrade produces a silent quality collapse, because the two halves now live in different vector spaces and nearest-neighbour search returns noise.
Domain vocabulary is the other trap. A general embedding model handles Hindi and English reasonably and Hinglish unevenly, because code-mixed text is under-represented in training. Product codes, model numbers and Indian legal citations are close to meaningless as vectors. This is why vector-only retrieval fails on the questions support teams actually receive, and why the next section exists.
Hybrid retrieval, because exact strings still matter
Lexical search, typically BM25, scores documents by term overlap with the query, weighted by rarity. It is old, cheap and unbeatable on exact strings: an invoice number, a SKU, a section reference, a surname, a pincode. Vector search is unbeatable on paraphrase and on questions phrased differently from the source. Combining them covers both, and the combination is usually a bigger improvement than any single model upgrade.
The standard combination is score fusion rather than a single blended score, because BM25 scores and cosine similarities are not on the same scale. Reciprocal rank fusion is the simplest method: take each result list, score every document by one over a constant plus its rank, and sum across lists. It requires no training, no tuning and no assumption about score distributions, and it works well enough that more elaborate schemes rarely justify their complexity.
For Indian corpora, hybrid retrieval is not optional. A query like HSN 6109 ka rate kya hai mixes a code, a Hindi particle and a Roman-script spelling. The vector side handles the intent, the lexical side catches the code, and neither alone gets both. Add a normalisation step for currency formats, Indian digit grouping, and the various spellings of the same city, and the retrieval quality moves again.
Reranking is the cheapest quality win
First-stage retrieval is optimised for speed across millions of chunks, so it returns a candidate list that is roughly right. A reranker is a slower, more accurate model that reads the query and each candidate together and scores relevance directly. It only has to look at fifty documents, so it can afford to be expensive, and it typically produces a meaningful lift in the top few results.
The pattern is retrieve wide, rerank narrow. Pull fifty to a hundred candidates with hybrid search, rerank them with a cross-encoder or a small language model prompted to score relevance, keep the top five to eight above a threshold, and pass only those to the generator. Fewer, better passages reduce cost, reduce distraction and make citation accurate, because there are fewer candidates for the model to misattribute.
The threshold matters as much as the model. If nothing clears it, the correct behaviour is to say the knowledge base does not cover this and offer a handoff, not to pass the least bad passage. A retrieval system without an abstain path will always produce an answer, and the answers it produces when the corpus is silent are the ones that damage trust most.
Grounding and citation are two different claims
Grounding means the answer is derived from the retrieved passages rather than from model memory. Citation means the answer names the passage it came from in a form a reader can check. A system can be grounded without citations, and it can produce citations that do not support the sentence, which is the more dangerous failure because it looks rigorous.
The practical control is span-level attribution. Ask the model to quote the sentence it relied on, then verify mechanically that the quote appears in the retrieved text, then render the citation from that verified span. This catches the common case where the model writes a plausible sentence and attaches the nearest document title. It does not catch a correct quote used to support the wrong conclusion, which is why the eval set needs a human review of reasoning, not just of string matching.
Citation format matters for trust. Link to the specific section, show the document title and the last updated date, and make the passage visible on hover or expand. A footnote that says source: internal documentation is worse than no citation, because it implies a verification that did not happen. If the answer synthesises three passages, cite three, and accept that the interface gets busier.
RAG does not fix bad source content
This is the most expensive misconception in the field. If the knowledge base says delivery takes five to seven days and the operation now takes ten, RAG will retrieve that passage and the model will state five to seven days with a citation. The system has worked exactly as designed. The failure is upstream, and no amount of prompt engineering, reranking or model upgrade will invent the correct number.
Worse, RAG amplifies bad content. A vague policy becomes a vague answer delivered with confidence. Two contradictory pages become an answer that changes depending on which chunk ranked higher that day. A page written for a different audience becomes an answer that is technically true and contextually wrong. Without retrieval, the model might have said it did not know. With retrieval, it has a document that appears to authorise the claim.
The remedy is content operations, not machine learning. Assign an owner to every indexed source, record a last-reviewed date, expire documents that pass it, and treat the index as a product with a changelog. Run a weekly report of the most-retrieved passages and read them, because the passages the system leans on are the ones that most need to be correct. This work is unglamorous and it is the actual project.
Evaluating retrieval before you evaluate answers
Split the eval. Retrieval metrics ask whether the right passage was in the candidate set: recall at k, and mean reciprocal rank for where it landed. Answer metrics ask whether the generated text is supported by what was retrieved and whether it answers the question. If recall at ten is eighty percent, the remaining twenty percent of failures are not the model’s fault and no prompt change will fix them.
Build a set of perhaps a hundred real questions with the correct source passage labelled by a human. That is a day of work and it is the only way to compare two chunking strategies, two embedding models or two rerankers without arguing. Re-run it on every change to the index, because a re-chunk silently changes what is retrievable and the regression will otherwise appear as a vague complaint three weeks later.
For faithfulness, use a second model to check whether each sentence of the answer is entailed by the cited passage, and sample the disagreements for human review. This is approximate and it catches the blatant cases, which are most of them. For answer correctness, use humans on a sample. The temptation to automate the whole evaluation is strong and it produces a metric that measures the judge model rather than the product.
Freshness, permissions and the Indian corpus
An index is a copy, and a copy inherits every obligation of the original. If a customer exercises erasure, the vector store must be scrubbed, not just the database. If a document is confidential to a team, the retrieval step must filter by the caller’s entitlements before the model sees anything, because a prompt cannot be trusted to keep a secret it has already been given. Filtering after generation is not a control; the model has already read the text.
Freshness is a scheduling problem with a quality symptom. Incremental indexing on write is better than a nightly rebuild, but deletions are the part everyone forgets. A document removed from the source but still in the index will be cited confidently for months. Tombstone the deletion, purge the vectors, and keep the purge in the same transaction boundary as the source delete where possible.
The Indian corpus adds specific work. Many PDFs are scanned images with no text layer, so optical character recognition is a prerequisite and its errors become retrieval errors. Legacy Hindi fonts produce mojibake unless handled. The same content may exist in Devanagari and Roman script, which are different strings and different vectors unless you normalise or index both. And support conversations are Hinglish, which means the query language and the document language rarely match, so cross-lingual retrieval is the default case rather than an edge case.
Worked example: a support agent over four thousand documents
Take a Growth customer with 5M AI tokens a month included, paying ₹7,999 plus 18% GST. The support team has four thousand help articles, product manuals and policy pages, and wants an agent that answers on WhatsApp. The naive build embeds whole pages, retrieves the top three, and pastes them into a prompt. Answers are sometimes right, sometimes cite a 2022 policy, and the team concludes that the model is not good enough.
The rebuild changes four things. Chunking by heading with the heading path prepended, which makes each fragment self-describing. Hybrid retrieval with BM25 alongside vectors, which fixes the SKU and invoice-number questions that vector search never got. A reranker over the top fifty, which lifts the correct passage into the top five. And a freshness job that expires anything unreviewed for twelve months, which removes the 2022 policy from the index entirely.
The cost effect is worth noting. Two hundred conversations at five turns each is a thousand turns, and at roughly two thousand tokens per turn of context and output that is about two million tokens, comfortably inside the allowance. Trimming the context from eight passages to three cuts the per-turn tokens by more than half, which either halves the bill at ₹0.30 per thousand tokens or buys twice the conversation volume for the same money. Better retrieval is also cheaper retrieval, because precision means you can send less.
Frequently asked questions
Is RAG better than fine-tuning?
They solve different problems. RAG is better for facts that change, for content you did not train on, and for anything that needs a citation. Fine-tuning is better for behaviour: tone, format, classification and task adherence. If your problem is that the model does not know your refund policy, that is retrieval. If the problem is that it writes in the wrong register, that is tuning.
How large should chunks be?
Start between two hundred and five hundred tokens with a small overlap, and split on document structure rather than a fixed character count. FAQs want smaller chunks, narrative wants larger ones, and tables need the header repeated. The test is whether a chunk still makes sense when shown alone, because that is how the model will see it.
Why does my RAG system cite the wrong document?
Usually because the model is citing the top-ranked result rather than the passage it actually used. Ask for a quoted span, verify the quote appears in the retrieved text, and render the citation from the verified span. If the quote check fails often, the passages are too long or too numerous, and the model is guessing which one supports its sentence.
Do I need a vector database?
Not necessarily. A vector index inside Postgres is sufficient for tens of thousands to low millions of chunks and avoids a second system to secure, back up and erase from. A dedicated vector store earns its place when you need very large scale, heavy filtering or low-latency approximate search. Start with what you already operate.
Can RAG work in Hindi and Hinglish?
Yes, with extra work. Use an embedding model that handles Indic languages, index both Devanagari and Roman transliterations where they exist, and keep a lexical index because code-mixed queries often contain exact strings. Test with real queries from your support logs rather than translated English ones, since the phrasing differs substantially.
How do I stop the model answering when the corpus has nothing?
Add an abstain path. Rerank candidates, apply a relevance threshold, and if nothing clears it, return a scripted message that says the knowledge base does not cover this and offers a handoff. Evaluate the abstain behaviour explicitly, because a system tuned only for answer rate will learn to answer everything, which is the failure you are trying to avoid.
Does RAG reduce hallucination?
It reduces one kind, the kind caused by missing facts. It does not eliminate hallucination, because a model can still misread a passage, blend two sources, or attach a real quote to a wrong conclusion. Grounding plus span verification plus a faithfulness eval gets you a meaningful reduction. A claim of zero hallucination is not a claim anyone can support.
How often should the index be rebuilt?
Index on write where you can, and run a full reconciliation weekly. The important part is deletion: a removed document must leave the index in the same window, or it will keep being cited. Track a last-indexed timestamp per source and alert when a source has not reported in longer than its expected change cadence.
What does DPDP mean for a RAG index?
The index is a copy of personal data, so purpose, retention and erasure all apply to it. Erasing a customer means removing their passages and vectors, not only the source row. Retrieval must also filter by the caller’s entitlements before generation, because once the passage is in the prompt, the disclosure has already happened.
How do I know retrieval is the problem and not the model?
Measure recall at k on a labelled set. If the correct passage is in the top ten less than about ninety percent of the time, fix retrieval first. If it is there and the answer is still wrong, the problem is generation, prompting or the passage itself. Without that split, every failure gets blamed on the model and every fix is a guess.
Sources and further reading
- Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Cormack, Clarke, Buettcher — Reciprocal rank fusion
- Nogueira, Cho — Passage re-ranking with BERT
- Es et al. — RAGAS: automated evaluation of retrieval augmented generation
- MeitY — Digital Personal Data Protection Act 2023