AI systems fail quietly at the retrieval layer
Pull-quote: “A retriever that misses produces the same output shape as one that works. That is the entire problem.”
Retrieval failures are silent because generation never fails loudly: the model writes fluent, confident text from whatever context arrived, so a document that was never retrieved surfaces as a wrong answer at the last stage rather than as an error where it happened. Measure the retriever on its own and much of what your users call hallucination turns into a recall number with a known fix.
This post is for anyone operating a retrieval-augmented system that demos well and produces complaints in production. It covers scoring retrieval independently of generation, the four places vector search alone breaks, how hybrid retrieval merges two rankings that share no unit, when a reranker earns its latency, and how permission filtering changes the answer. It assumes a basic pipeline: chunk, embed, store, retrieve, prompt.
One term does the most work here. Recall at k is the share of documents that should have been retrieved which appear in the top k results. It tells you whether the generator ever had a chance.
What you will be able to do
- Tell a retrieval failure from a generation failure using evidence rather than intuition.
- Build a labelled query set and report recall at k, mean reciprocal rank, and normalised discounted cumulative gain on the retriever alone.
- Recognise the four ways vector search fails without raising an error, including the filtered query that silently returns four rows.
- Fuse lexical and dense rankings by rank instead of by score, and say why score fusion is fragile.
- Map a measured failure signature to the change that fixes it.
Why does a retrieval failure look like a model failure?
Because the generator receives no signal that anything is missing, so it produces a complete, confident answer from an incomplete context and the defect appears at the last stage.
Two symptoms dominate. The first is a confident wrong answer assembled from passages that were topically close and factually beside the point. The second is a refusal: the system says it has no information on a subject sitting in the index, never retrieved. The refusal costs more, because users conclude the content is not there and stop asking, which removes the complaint that would have told you.
Inside an agent loop, neither symptom reaches a human. The context feeds a plan, the plan feeds a tool call, and the run reports success: the hallucinated success class from the agent failure taxonomy, entering through the retrieval layer instead of a tool.
The instrumentation that makes this diagnosable is small and almost nobody has it: log every retrieved chunk id, its score, and which retriever produced it, keyed to the answer returned. Without that record, every complaint is unfalsifiable and every fix is a guess.
How do you measure retrieval separately from generation?
Build a labelled set of real queries paired with the passages a knowledgeable person would cite, then score the retriever before the generator sees it.
| Metric | What it answers | What it catches |
|---|---|---|
| Recall at k | Did the needed passage make the top k | Coverage failures, where the answer never had a chance |
| Mean reciprocal rank | How high was the first relevant result | Ranking failures on questions with one right answer |
| Normalised discounted cumulative gain at 10 | Are the most relevant items at the top | Ranking failures when several passages matter |
| Recall at 100 minus recall at 10 | How much a reranker could recover | Whether reranking is worth its latency |
Harvest queries from logs rather than inventing them, aim for a few hundred, and keep the hard ones. Anthropic reported its retrieval work using one minus recall at 20, the share of relevant documents that failed to reach the top 20, which names the failure more directly than accuracy does.
One checkpoint proves the harness works. Swap in a retriever that returns random passages. Retrieval metrics should collapse to near zero while the end-to-end generation score moves far less than you expect. If both fall together, your generation eval is measuring retrieval and you cannot tell the two apart. That experiment is worth more than a week of prompt tuning, and it fits the discipline in grading the run rather than the reply.
Where does vector search alone actually break?
On exact identifiers, on filtered queries against an approximate index, on chunks that lost the context that made them meaningful, and on questions whose answer spans records.
Exact tokens. Ask an embedding model for “Error code TS-999” and it returns passages about error codes in general, because the embedding encodes meaning and the identifier carries almost none. A lexical index matches the literal string. Anthropic’s worked example generalises to part numbers, contract clauses, ticket ids, and drug names.
Filtered queries against an approximate index. This one surprises experienced teams. Vector indexes are approximate by design, and filtering commonly runs after the index scan rather than during it. The pgvector documentation states the consequence plainly: with HNSW and the default hnsw.ef_search of 40, a filter matching 10% of rows returns about four results on average. No error is raised. The result set is short, and the generator answers from four passages as happily as from forty.
-- HNSW is approximate and the filter runs after the scan
SET hnsw.ef_search = 100; -- default is 40
SET hnsw.iterative_scan = strict_order; -- pgvector 0.8.0 and later
SET hnsw.max_scan_tuples = 20000;Chunks stripped of context. A passage reading “the company’s revenue grew by 3% over the previous quarter” names no company and no quarter, so it is nearly unretrievable and misleading if retrieved.
Answers that span records. When the fact you need exists only as a relationship across documents, no chunk contains it and no ranking can surface it.
What does hybrid retrieval combine, and how are the two rankings merged?
A lexical ranking and a dense ranking, merged by rank position rather than by score, because the two scoring scales share no unit and no stable range.
Okapi BM25 scores a document from term frequency, damped by a saturation function, adjusted for document length and term rarity. Lucene’s implementation, which Elasticsearch and OpenSearch run, defaults to k1 = 1.2 and b = 0.75. Those constants control frequency saturation and length normalisation, and most teams should leave them alone until a measurement says otherwise.
Reciprocal rank fusion merges result sets without needing them to be comparable. For each document, sum one divided by a constant plus its rank in each list:
score = 0.0
for q in queries:
if d in result(q):
score += 1.0 / (k + rank(result(q), d))The constant k decides how much influence lower-ranked documents carry, and Elasticsearch defaults it to 60, following the 2009 paper by Cormack, Clarke, and Buettcher. A higher constant flattens the advantage of the top few positions. The method needs no tuning and no shared score scale, which is why it beats the alternative: min-max normalising cosine similarity and BM25 per query and adding them makes the blend depend on the score spread within each query, so the weighting changes from request to request and nobody can explain why.
When is reranking worth the extra latency?
When recall at 100 is much higher than recall at 10, because that gap is what a reranker recovers and nothing cheaper will.
The mechanism explains the gap. A bi-encoder embeds query and document separately, which makes a pre-built index possible and also limits it: the document vector was computed without seeing your question. A cross-encoder scores query and document together, far more accurately and far too expensively to run across a corpus. So you retrieve wide with the cheap model and rerank narrow with the expensive one.
Anthropic published a configuration worth copying as a starting point: retrieve the top 150 candidates, rerank, keep the top 20 for the prompt. On their evaluation, contextual embeddings plus contextual BM25 cut the top-20 retrieval failure rate by 49%, from 5.7% to 2.9%, and adding the reranker took the reduction to 67%, reaching 1.9%.
Two cautions. Reranking adds a network round trip on the serving path, so measure the gap before paying for it. And more context is not automatically better: the same work found 20 chunks outperforming 5 and 10, while Lost in the Middle found accuracy degrading when the relevant passage sits mid-context. Ordering matters as much as quantity, a theme the context budget treats in full.
What do you do about the links a chunk index cannot represent?
Either put the missing context inside the chunk, or move the relationship into a structure that can hold it. Three moves, cheapest first:
- Contextualise the chunk before indexing. Generate 50 to 100 tokens of chunk-specific context from the whole document, prepend it, and index it in both the embedding store and the lexical index. Anthropic measured a 35% reduction in top-20 failures from contextual embeddings alone and 49% combined with contextual BM25, at a one-time cost of about $1.02 per million document tokens using prompt caching.
- Hydrate the parent. Retrieve at chunk granularity for precision, then return the enclosing section for context. This costs nothing but tokens and fixes a surprising share of “the answer was almost there” complaints.
- Model the relationship explicitly. When the answer is a path across records rather than a passage, no chunking strategy reaches it. That is the case for graph retrieval, covered in knowledge graphs as the memory layer agents need.
How does permission filtering change retrieval quality?
It changes the candidate pool, so a retriever that scored well in evaluation can lose recall in production without a line of code changing.
Two mechanisms do the damage. The first is post-filtering an approximate index, described above: entitlement applied after the scan turns a healthy candidate list into a short one. The second is that your index may not carry the entitlement at all. Databricks documents both halves: you cannot apply row-level security or column masks to a view, and you cannot create an AI Search index from a table that has row filters or column masks applied. So entitlement has to be modelled inside the index, either as an access-control field you pre-filter on or as a separate index per security boundary.
The measurement consequence is the one teams miss. Run your retrieval evaluation as a restricted user, not as an administrator. An eval with full permissions measures a system nobody uses, and it reports healthy recall right up to the day a scoped user complains.
Which retrieval architecture should you pick?
The one your measurements point at. Match the change to the signature in your numbers rather than picking from a list of named architectures.
| Signature in your numbers | Likely cause | What to change |
|---|---|---|
| Recall at 100 high, recall at 10 low | Ranking, not coverage | Add a cross-encoder reranker over a wide candidate set |
| Recall poor on identifier and code queries | No lexical path at all | Add BM25 and fuse by rank |
| Recall healthy, answers still wrong | Chunks lost the context that gave them meaning | Contextualise before indexing, or hydrate the parent |
| Recall poor only on multi-entity questions | The answer is a path, not a passage | Graph retrieval from resolved entities |
| Recall fine offline, poor in production | Permission filtering changed the candidate pool | Pre-filter on an access field, or index per boundary |
| The retrieved set is often irrelevant and the model uses it anyway | Nothing grades retrieval before generation | Score the retrieved set and re-query on a low grade |
Read that table as a diagnostic sequence, not a menu. Every row starts with a number you can produce this week, and none requires deciding in advance which retrieval architecture you are building.
Where this goes wrong
Retrieval quality is reported as a generation metric. The symptom is a single answer-quality score that never explains itself, because only the end of the pipeline is measured. Score the retriever separately and the same complaint becomes a recall number with known fixes.
Chunk size becomes the only lever. The symptom is weeks spent between 400 and 800 tokens with no durable improvement, from treating a chunking parameter as the whole design space. Chunk size matters far less than whether a lexical path exists and whether chunks carry their context.
A reranker is added before the gap is measured. The symptom is more latency and roughly the same answers, from adopting a technique rather than fixing a signature. Compare recall at 10 with recall at 100 first, because if they are close there is nothing to recover.
Nobody logs what was retrieved. The symptom is a backlog of quality complaints nobody can reproduce, because retrieved chunk ids were never written beside the answer. This is the cheapest item here and the one that unblocks the rest.
Common questions
Is retrieval still necessary now that context windows are large?
For a knowledge base under roughly 200,000 tokens, around 500 pages, Anthropic’s guidance is to put the whole thing in the prompt with prompt caching rather than building retrieval. Above that scale you need retrieval, and long contexts do not remove the ranking problem: accuracy still depends on where the relevant passage lands.
What is a good recall at k target?
There is no universal number, because it depends on how many passages your questions need and how tolerant the task is. Fix k at the number of chunks you actually pass to the model, track the metric over time, and treat a regression as a release blocker. Targets borrowed from someone else’s corpus tell you nothing.
Should I use a vector database or a search engine that does both?
If you need lexical and dense retrieval fused, a search engine implementing both plus rank fusion removes an integration from your system. If you already run Postgres and your corpus is modest, pgvector alongside full-text search is often enough. Decide on operational grounds, since the quality difference comes from hybrid plus reranking, not the vendor.
Does reranking remove the need for hybrid retrieval?
No, because a reranker only reorders what the first stage retrieved. If the exact-match candidate never entered the top 150, no reranker will find it. Hybrid retrieval fixes coverage and reranking fixes ordering, which is why the published gains stack rather than overlap.
Next steps
Pick 50 real questions from your logs, mark the passage that answers each, and compute recall at 10 and recall at 100 for your current retriever. Those two numbers tell you within an afternoon whether you have a coverage problem or a ranking problem, and the two have different fixes.
Then read knowledge graphs as the memory layer agents need if your gaps cluster on multi-entity questions, and agentic text-to-SQL if the same silent-failure pattern shows up over structured data. More engineering writing sits in the Signals index.
Measuring the retriever separately, and running that evaluation as a restricted user, is the retrieval discipline behind EvidAI in pharmaceutical evidence review, where a missed source is a finding rather than an inconvenience.