Deep Read, retrieval

Why Your AI Still Makes Things Up

Retrieval-augmented generation grounds a model in real documents, but only when the right one is found and actually read. Here is the pipeline, and every place it quietly breaks.

The Curio Team · 5 min readWritten with AI from the sources below and checked against them before publishing.

Ask a model a hard factual question and it answers with total confidence. Sometimes it's simply wrong. The words still sound right, because sounding right is the whole objective. A language model predicts the next plausible token given everything before it, never the true one, because it has no separate notion of truth. Retrieval-augmented generation exists to close that gap. We want to explain, honestly, why it helps and where it keeps letting you down.

Plausible text is not truth

Here's the core mismatch. A base model learned patterns from an enormous pile of text, so it's brilliant at continuing a sentence in a way that fits. Fit is not fact. When the training data is thin on your exact question, the model fills the hole with something statistically reasonable, and a reasonable-sounding wrong answer is what we call a hallucination. Anthropic's guidance on this is blunt. You cut fabrication by giving the model real source material and instructing it to answer only from that material.³

RAG is the standard way to hand over that source material. Instead of trusting the weights to remember every fact, we fetch relevant documents at query time and paste them into the prompt. The model then reads and paraphrases real text rather than reconstructing it from foggy memory. Done well, this is the gap between a student answering from an open book and one guessing on a closed exam. The open book wins.

The pipeline, stage by stage

The original RAG design paired a retriever with a generator, and every serious system since keeps the same skeleton. Five stages run in order. Each one can quietly fail, which matters, because the whole chain is only as strong as its weakest link.⁴

• Chunk. Long documents get split into passages small enough to embed and to cite later.
• Embed. Each chunk becomes a vector, a numeric fingerprint of its meaning.
• Retrieve. Your question is embedded too, and the nearest chunks by similarity come back.
• Augment. The top chunks are inserted into the prompt as working context.
• Generate. The model answers, grounded in the passages you handed it.

Chunk size is a real design choice, not a default to accept. Too small and a passage loses the context that made it meaningful. Too large and the retriever's aim gets blurry, because one vector now has to represent five different ideas. We tune it against real questions and re-check when the corpus changes, since the right size for a legal contract is not the right size for a chat log.

Embeddings are the quiet workhorse here. Similar meanings land near each other in vector space, so a question about refund windows retrieves the refund policy even when the wording differs. That's powerful and it's also where trouble starts. Similarity is not the same as relevance, and the model downstream can only use what the retriever chose to surface.²

One upgrade closes a lot of ground: rerank the results. First-pass vector search is fast but coarse, so it returns roughly-relevant chunks. A second model then scores those candidates against the actual question and reorders them, pushing the truly relevant passage to the top where the generator will read it. We think of retrieval and reranking as two different jobs, and conflating them is how good documents end up ranked eighth.

Where retrieval quietly breaks

Two conditions hide inside every grounded answer. Retrieval only helps if the right document is found, and then only if the model actually uses it. Both must hold. Miss either and you're back to a confident guess dressed up as a citation. We keep coming back to that pair, because almost every real failure is one of the two breaking, sometimes both at once on the same query.

Every hallucination in a RAG system traces back to one of a few points. Knowing them turns debugging from guesswork into a checklist. Here are the ones we hit most.

• Retrieval miss. The right passage exists but ranks below the cutoff, so the model never sees it.
• Chunking damage. A single fact gets split across two chunks, and neither one carries the full answer.
• Lost in the middle. Models attend less to material buried between long blocks, so a good passage gets skimmed over.
• Stale index. The source document changed but its embeddings didn't, so you ground the answer on yesterday's truth.
• Ignored evidence. The correct passage is right there in the prompt, and the model still answers from its own priors.

That last one stings the most. You did the retrieval work, the evidence is present, and the answer is still fabricated. The fix is prompt discipline. Tell the model to cite the passage it used, to quote before it claims, and to say I don't know when the context is silent. Anthropic's contextual retrieval work also shows a structural fix: prepend a short summary to each chunk so it keeps its meaning after being torn from its document.¹

Retrieval doesn't make a model honest. It gives an honest model something true to read.

What we'd actually measure

Grounding is testable, so test it. Separate the two questions. Did retrieval fetch the right document, and did the model use it faithfully? A wrong final answer with the correct passage in context is a generation problem. A wrong answer with the passage missing is a retrieval problem. Measure them apart and you know which knob to turn. We treat every hallucination as a signal, not a mystery, and it usually points straight at one broken stage.

One honest caveat. RAG grounds an answer in documents, but it can't verify the documents themselves. Feed it a wrong policy and it will faithfully cite the wrong policy. Garbage in, confident garbage out. So the corpus is part of the model now, and we curate it with the same care we'd give the prompt. Retrieval moves the trust problem. It does not delete it.

Fluency was never the hard part for these systems. Truth is. We build retrieval so the model has something real to be fluent about, and then we check, relentlessly, that it read the thing we gave it. Get both right and the confident voice finally has earned its confidence.