What retrieval-augmented generation actually gets wrong

Retrieval-augmented generation sounds simple: embed your documents, store the vectors, retrieve the closest matches, hand them to a model, get a grounded answer. The pitch is that the model can't hallucinate if it's just reading your own content back to you. In practice, most RAG systems that get built this way work great in the demo and degrade steadily as the document set grows and gets messier. The failure isn't in the model. It's almost always upstream, in retrieval, and it's rarely obvious until someone asks a question that exposes it in front of a customer instead of in testing.

This matters because RAG has quietly become the default architecture for anything described as 'let the AI answer questions from our docs.' Support tools, internal knowledge bases, contract search, onboarding assistants: almost all of them are some version of retrieve-then-generate. Which means almost all of them inherit the same handful of failure modes, whether the team building them knows it or not.

Chunking is where it starts going wrong

The default move is to split documents into fixed-size chunks, embed each one, and call it done. This works until a chunk boundary lands in the middle of a table, a clause, or a sentence that only makes sense with the paragraph before it. The model then answers confidently from a fragment that's missing its own context. We've seen pricing tables split so that the number retrieved belongs to a different row than the label next to it. Nobody catches this in testing because the ten sample documents used for the demo happen not to have that problem.

The insidious part is that a badly chunked document doesn't fail loudly. It fails by producing an answer that sounds completely reasonable and is quietly wrong, sourced from a fragment that was never meant to stand alone. A support agent trusting that answer has no way to know it's wrong without going and reading the original document themselves, which defeats the point of building the system in the first place.

• Fixed-size chunking ignores document structure, headers, tables, and lists get sliced arbitrarily.
• Overlapping chunks reduce boundary loss but multiply storage and retrieval noise.
• Semantic chunking, splitting on meaning instead of character count, is more expensive to build but survives real documents.
• Metadata gets dropped at chunk time, so retrieval loses the source, date, or section a chunk came from.

Similarity search finds close, not correct

Vector search returns the chunks that are semantically nearest to the query. Nearest is not the same as most relevant, and it's definitely not the same as correct. A query about last quarter's return policy can retrieve three chunks that are all about return policy but from three different years, and the model will happily blend them into an answer that was never true at any point in time. This is the single most common RAG bug we see in production: the system retrieves plausible-sounding text and treats plausibility as accuracy.

The problem compounds with document volume. At ten documents, the nearest match is usually also the right match, because there's little else competing for that semantic space. At ten thousand documents, especially ones that update, get superseded, or contradict earlier versions, near misses become common, and the system has no innate sense of recency or authority. It doesn't know that the March policy replaced the January one unless you tell it, explicitly, as part of the retrieval logic rather than hoping the embedding captures it.

The fixes that actually move the needle

Hybrid search, combining keyword and vector retrieval, catches exact terms that embeddings blur past, like SKUs, case numbers, or names. Reranking the top candidates with a second, more precise pass before they ever reach the model cuts down on near-miss chunks getting used. And setting a hard relevance floor, so the system says 'I don't have enough information' instead of forcing an answer from weak matches, is the single change that does the most for trust. None of this is exotic. It's just work that doesn't show up in a weekend demo, and it's the difference between a RAG system people use and one they quietly stop trusting after the third wrong answer.

Filtering by metadata before the similarity search runs, rather than after, is another underused fix. If you know the query is about a specific product line or a specific date range, narrow the candidate pool first. It's a small architectural decision that removes an entire category of wrong-but-plausible retrievals before they ever get a chance to compete for the top slots.

A retrieval system that can't tell you when it found nothing good will always tell you something.

Test it the way it'll actually get used

The gap between a RAG demo and a RAG system that holds up is almost entirely a testing gap. Demos get tested with questions the builder already knows the answer to, phrased the way the builder would phrase them. Real usage includes vague questions, questions about things that aren't in the documents at all, and questions phrased in ways nobody anticipated. If your test set doesn't include at least as many 'this should return nothing useful' cases as 'this should return a clean answer' cases, you haven't tested the part that actually determines whether people trust the system.

If you're evaluating a RAG build, ask to see it fail. Ask what happens when the answer isn't in the documents. If the demo can't show you a clean 'I don't know,' the system hasn't been tested against the case that actually matters, and it will eventually answer a question it had no business answering, with total confidence, in front of the person you least wanted it to.

Freshness is a retrieval problem too, not just a data problem

Teams often treat 'keep the index up to date' as a background sync job and move on, but stale retrieval is a distinct failure mode from bad chunking or weak similarity search. A policy document gets updated, the old version is still in the index next to the new one, and nothing in the embedding tells the system which one is current. Without an explicit versioning scheme, superseded documents removed or clearly flagged rather than left to compete on equal footing, the system will occasionally retrieve the outdated version and present it with the same confidence as the current one. This is worse than a stale cache in a normal application, because there's no error, no visible staleness indicator, just a wrong answer that reads exactly like a right one.

The practical fix is boring but effective: treat document ingestion as a real pipeline with versioning, not a one-time load. When a document is replaced, the old chunks get removed from the index, not just superseded in a source folder somewhere. When a document is deleted, its chunks go with it. It sounds obvious written down, and it's the single most common gap we find when auditing an existing RAG system that's been live for more than a few months.