Your RAG agent returns wrong answers - fix the retrieval

Your RAG agent confidently returns an answer that is wrong, out of date, or about the wrong customer entirely - and the instinct is to blame the model or reach for a bigger one. That is almost always the wrong move. A retrieval-augmented agent answers from the passages it was given, so when the answer is wrong, the usual cause is that retrieval handed it the wrong context. The model did its job faithfully on bad input. Fix retrieval and the same model starts answering correctly.

This is the troubleshooting runbook we use when a production RAG system gives wrong answers. The five retrieval failure modes, how to tell which one you have, and the specific fix for each - written for UK and US teams running a RAG agent that worked in the demo and is now embarrassing them in production.

Wrong answers are usually a retrieval problem, not a model problem

Before changing anything, run one diagnostic: for each wrong answer, log exactly what passages retrieval returned. This single step tells you where the bug lives. If the correct passage is not in the retrieved set, no model can answer correctly - it never saw the right information, so this is a retrieval bug and a bigger model will not save you. If the correct passage is in the set but the model still answered wrong, that is a generation or prompting issue, closer to the territory of handling AI hallucinations in production. In practice, the large majority of "wrong answer" tickets are the first kind. The rest of this runbook assumes you have confirmed the right context is not being retrieved.

The five retrieval failure modes

1. Bad chunking

Chunking is the highest-leverage and most common culprit. If documents are split into fixed-size blocks with no regard for structure, a single answer gets cut across two chunks, a table is severed from its heading, or a chunk mixes three unrelated topics so its embedding represents none of them well. Retrieval then either misses the passage or returns a fragment without the context that makes it usable. The fix is to chunk on structure, not character count: split on headings, sections and natural boundaries, keep tables and lists intact, and add a little overlap so a thought is not lost at a boundary. The mechanics of good chunking are covered in how RAG works - architecture, chunking and retrieval explained; if you only fix one thing, fix this.

2. No reranker - you trust raw vector similarity

Vector search is fast and approximate. It finds passages whose embeddings are close to the query, but closeness is not the same as relevance - the top result by cosine similarity is often not the most useful passage for the question. Teams that feed the raw top results straight into the model accept whatever the embedding happened to rank first. The fix is a reranker: retrieve a wider candidate set, say the top twenty, then run a cross-encoder reranker that scores each candidate against the actual query and reorders them, so the genuinely most relevant passages rise to the top before the model sees them. Adding a reranker is one of the highest-impact accuracy upgrades for a RAG system that already retrieves roughly the right neighbourhood.

3. A stale or incomplete index

If the agent answers as if it is reading an old version of reality, the index is the suspect. RAG only knows what has been embedded and indexed - if documents changed after indexing, or new documents were never ingested, retrieval cannot return what is not there, and the agent answers from outdated content with full confidence. The fix is an ingestion pipeline you can trust: re-index on a schedule or on document change, track which version of each source is indexed, and monitor for ingestion failures so a silently-dropped document does not become a class of wrong answers. Many "the agent is wrong" reports are simply "the agent is reading last quarter's document."

4. Missing metadata filters

This is the failure mode that produces the most alarming answers - the agent returns correct information about the wrong thing. It pulls another customer's record, a different product's spec, or last year's policy, because vector similarity alone has no notion of which customer, product or date the query is about. The fix is metadata filtering: tag every chunk at ingestion with the attributes that matter - customer ID, product, document type, effective date, access level - and filter on them before or during vector search so retrieval can only return passages from the correct scope. This is also a security and data-isolation control, not just an accuracy one, which makes it non-optional for any multi-tenant agent.

5. The wrong top-k and context packing

The last mode is feeding the model too little or too much. Too few passages and the answer is not in context; too many and the relevant passage is buried in noise the model dilutes or ignores, especially in the middle of a long context window. Either way the answer suffers. The fix is to tune top-k to the question type and the model's context budget, put the highest-ranked passages where the model attends best, and trim low-scoring candidates rather than padding the prompt. More context is not better context - the goal is the right passages, clearly placed, and nothing competing with them.

How to debug retrieval methodically

Work it in order so you fix the real cause, not a symptom. Build a small evaluation set of real questions with their known-correct source passages - twenty to fifty is enough to start. For each, log the retrieved set and check whether the correct passage is present and where it ranks. If it is absent, the problem is chunking, indexing or filtering - check those first. If it is present but ranked low, you need a reranker or better top-k handling. Re-run the set after each change so you can see the hit rate move, rather than guessing. This retrieval-quality metric, not vibes, is what tells you the system improved - and it is the same evaluation discipline that keeps a production agent honest as data and usage grow, alongside watching what it all costs to run a RAG app at a million queries a month.

Where retrieval quality fits the bigger build

A RAG agent is only as good as the context it retrieves, so retrieval quality is the part of the build that deserves the most engineering attention and the most honest measurement. Getting chunking, reranking, indexing and filtering right is core to any production AI agent development engagement, and the surrounding evaluation and data pipeline are where AI development services earn their place. Fix the five failure modes, measure retrieval against a real eval set, and the same model that was embarrassing you starts answering from the right context every time.

FAQ

Why does my RAG agent give wrong answers even with a good model?

Because the model answers from whatever passages retrieval gives it. If retrieval returns the wrong, stale or incomplete context, a strong model will faithfully produce a wrong answer. Log the retrieved set for each bad answer - if the correct passage is not in it, the bug is in retrieval, and upgrading the model will not fix it.

What is a reranker and do I need one?

A reranker is a second-stage model, usually a cross-encoder, that scores each retrieved candidate against the actual query and reorders them so the most relevant passages come first. Vector similarity finds the right neighbourhood but does not rank true relevance well, so a reranker over a wider candidate set is one of the highest-impact accuracy improvements for most RAG systems.

Why does my RAG agent return information about the wrong customer or product?

Vector similarity has no built-in notion of scope, so without metadata filtering it can return a passage from any customer, product or time period that happens to be semantically close. Tag each chunk with attributes like customer ID, product and date at ingestion, and filter on them before or during retrieval so the agent can only see passages from the correct scope. It is an accuracy fix and a data-isolation control.

How do I know if chunking or indexing is the problem?

Build a small set of real questions with their known-correct source passages and log what retrieval returns for each. If the correct passage never appears in the retrieved set, suspect chunking (the passage was split or diluted) or indexing (it was never embedded or is out of date). If it appears but ranks low, the issue is ranking - add a reranker or adjust top-k.

The bottom line: when a RAG agent answers wrong, look at retrieval before the model. Log what was retrieved, then work the five failure modes - chunk on structure, add a reranker, keep the index fresh, filter by metadata, and tune top-k. Measure each change against a real evaluation set so you know retrieval actually improved. Do that and you fix the wrong answers at their source, with the model you already have.