Your RAG agent keeps making things up. Here is why

Your retrieval-augmented agent was supposed to end hallucinations - ground every answer in your own documents, no more confident nonsense. Then it cites a policy you never wrote, quotes a price that does not exist, and does it with total conviction. The instinct is to blame the model and reach for a bigger one or a fine-tune. That is almost always the wrong fix, because the model is rarely where a RAG agent breaks. It hallucinates because the retrieval step failed quietly and handed it garbage - and a model given garbage will fill the gap every time.

The model is not the problem - retrieval is

A RAG system has two halves: retrieval finds the relevant passages, and generation writes an answer from them. When the answer is wrong, teams stare at the generation half because that is where the words came from. But the model can only work with what retrieval fed it. If retrieval returned the wrong chunk, half the right chunk, or nothing useful, the model does the human-sounding thing and improvises to sound helpful. The hallucination is real; its cause is upstream. So the debugging question is never "why did the model lie?" - it is "what did we actually retrieve, and was the answer even in it?" Log the retrieved chunks for a failing question and the culprit is usually obvious in seconds.

The three retrieval failures that cause it

Almost every made-up RAG answer traces back to one of three upstream failures. Fix these before you touch the model.

• Bad chunking. If you split documents by a fixed character count, you slice sentences and tables in half and scatter one answer across two chunks. Retrieval then fetches a fragment that looks relevant but is missing the half that matters, and the model guesses the rest. Chunk on meaning - sections, paragraphs, logical units - so a retrieved chunk is a complete thought, not a random 500-character window.
• Weak search. If your embeddings are poor, or you rely on pure keyword match, the genuinely relevant passage never surfaces in the top results. The model only ever sees near-misses and answers from those. This is a retrieval-quality problem, and it rests on the same foundations we cover in what a vector database is and why it matters - good embeddings and a search that actually ranks meaning.
• No reranking. Vector search is fast but approximate; the single best passage is often sitting at result 8, not result 1. If you feed the model only the top few by raw similarity, the real answer never makes it into the prompt. A reranker re-scores the top candidates for true relevance so the best chunk is the one the model reads.

Grounding: the safety net when retrieval still misses

Even with good retrieval, sometimes the answer simply is not in your documents. That is the moment a RAG agent is most dangerous, because its default is to answer anyway. Grounding is the discipline that stops it. Two parts do most of the work.

• Answer only from the sources. Instruct the model, and verify, that it uses the retrieved passages and nothing from its own training. Ask it to cite which chunk each claim came from. A claim that cannot point to a source is the claim most likely to be invented, and now you can catch it.
• Let it say "I do not know". When retrieval returns nothing relevant, the correct answer is to admit it and offer a next step - not to improvise. An agent that says "I could not find that, let me connect you to someone" builds far more trust than one that confidently makes something up. This matters even more once the agent can take actions, which is why we build every production AI agent to fail loudly rather than guess.

Fix the pipeline, not the model

The order of work is what saves you time and money. Start by logging retrieval for your worst questions and confirm whether the answer was even present in what you fetched - nine times out of ten it was not, and that tells you exactly where to look. Improve chunking so passages are whole, upgrade the embeddings and search so the right passage ranks, add a reranker so it lands in the prompt, and only then wrap the whole thing in grounding so an empty retrieval produces an honest "I do not know" instead of a guess. Choosing between this and training a model is its own decision, and we lay out the trade-offs in fine-tuning versus RAG; for hallucinations specifically, retrieval is almost always the cheaper and more durable fix, and it is the heart of the AI systems we build.

Frequently asked questions

Will a bigger or better model stop the hallucinations?

Usually not. If the cause is retrieval handing the model the wrong context, a stronger model just writes a more convincing wrong answer. A better model can help at the margins once retrieval is solid, but spending there first is treating the symptom. Fix what the model is given before you upgrade the model.

How do I even know it is a retrieval problem?

Log the chunks your system retrieved for a question it got wrong and read them. If the correct answer is not in those passages, retrieval failed and generation never had a chance. If the answer is there and the model still got it wrong, then the problem is in the prompt or the generation step. That one check settles most RAG debugging.

What is reranking and do I need it?

Reranking is a second pass that re-scores your top retrieved candidates for genuine relevance, rather than trusting raw vector-similarity order. You need it when the right passage is being retrieved but sits below the cut-off you feed the model, so it never reaches the prompt. On non-trivial document sets it is one of the highest-return additions you can make.

Can grounding alone stop hallucinations without fixing retrieval?

It reduces the damage but does not fix the cause. Grounding makes the agent admit when it has nothing and cite where claims come from, which stops confident invention. But if retrieval keeps missing the right passage, a well-grounded agent will simply say "I do not know" to questions it should have answered. You want both - good retrieval so it can answer, and grounding so it is honest when it cannot.

The short version: a RAG agent that makes things up is almost always a retrieval failure wearing a generation costume. Log what you actually fetched, fix chunking so answers stay whole, strengthen search and add a reranker so the right passage reaches the model, then ground the output so an empty result is an honest "I do not know". Do that and the hallucinations fall away - without the cost and delay of chasing the model that was never really the problem.