RAG Failure Triage: Separating Retrieval Misses From Generation Errors

End-to-end accuracy is a product of stage rates

For a question to be answered correctly from retrieved context, four things must all happen: a document that supports the answer exists in the index, the retriever puts it in the top k candidates, the packing step (reranking, deduplication, truncation to the token budget) keeps it in the prompt, and the generator reads it correctly. If each step is roughly independent, the success rate is the product of the conditional rates:

P(correct from context) = P(doc exists) x P(in top-k | exists) x P(in prompt | top-k) x P(right | in prompt)

example (illustrative): 0.97 x 0.61 x 1.00 x 0.90 = 0.53
after fixing retrieval: 0.97 x 0.95 x 0.85 x 0.90 = 0.70
after fixing packing:   0.97 x 0.95 x 0.98 x 0.90 = 0.81

Two consequences follow. First, the stage with the lowest conditional rate caps the system, and improving any other stage gives proportionally little. Moving the generator from 0.90 to 0.95 in the first line adds under three points; fixing retrieval adds seventeen. Second, fixing one stage often exposes the next. Widening k from 4 to 30 in the example raises top-k recall but, without a reranker, many of the newly found passages sit at ranks 5 to 30 and never reach a 4-passage prompt. The packing stage that looked perfect was only perfect because nothing reached it. The model generator also answers some questions from its own parametric knowledge without the right context, which adds a little on top of the product and hides some retrieval misses in aggregate numbers.

A taxonomy that maps to fixes

Barnett and colleagues' study of RAG systems lists seven failure points; for triage, it helps to collapse them into classes that each have a distinct owner and fix:

The last row matters more than teams expect. In practice, a meaningful share of "failures" in a new evaluation set are label problems: an outdated reference answer, a question with two valid sources where only one is marked as gold, or a judge that penalises a correct answer phrased differently. Triage that skips this class ends up tuning the pipeline to match label noise.

The logging that makes triage possible

You can only attribute a failure to a stage if every stage's output is recorded per request. The minimum trace is short, but each field has to reflect what actually happened, not what was configured:

{
  "request_id": "...",
  "query": "original user text",
  "rewritten_query": "standalone query actually searched",
  "candidates": [{"doc_id": "d42#3", "score": 0.71, "rank": 1}, ...],   // all k, first stage
  "reranked": [{"doc_id": "d17#1", "score": 4.2}, ...],                 // if a reranker ran
  "prompt_passages": ["d17#1", "d42#3", "d08#2"],                       // after truncation
  "prompt_tokens": 3910, "truncated": true,
  "answer": "...", "model": "generator-version", "index_version": "2026-10-28"
}

The prompt_passages field is the one most often wrong. Many stacks log the retrieved list and assume all of it reached the model, while a template change, a longer conversation history or a token limit truncated the context downstream. Log the passage identifiers from the final assembled prompt. Tracing tools built on OpenTelemetry conventions (for example OpenInference with Arize Phoenix) or LangSmith can capture these spans automatically when the framework is instrumented; check what your version records for retriever and prompt-assembly steps before relying on it.

The triage procedure

With a labelled set (questions, reference answers and the identifiers of passages that support them), each failure can be classified mechanically, in stage order, so that it is attributed to the earliest stage that lost it:

def triage(case, trace):
    gold = set(case.supporting_passages)            # may be several valid sources
    if not gold or not any(index.contains(g) for g in gold):
        return "coverage_gap"
    if not gold & {c.doc_id for c in trace.candidates}:
        return "retrieval_miss"
    if not gold & set(trace.prompt_passages):
        return "ranked_or_packed_out"
    if judge.answer_is_correct(case, trace.answer):
        return "evaluation_error"                  # verify by hand: label or judge problem
    return "generation_error"

Run it over every failure in the evaluation set and sort by class. The largest bucket is the next project. Three refinements make it robust. Allow several gold passages per question, otherwise a retriever that found a different valid source is counted as a miss. Record the rank of the best gold passage even on success, because a pipeline where gold sits at rank k is one corpus change away from failure. And split generation errors further when the bucket is large: unsupported claims (the model ignored context), misreads (wrong detail from the right passage) and incomplete answers respond to different fixes.

Judges, labels and how sure you can be

Labelled gold passages are expensive, so many teams use an LLM judge to score context relevance and faithfulness on unlabelled traffic, as in the RAGAS and ARES frameworks and the "RAG triad" of context relevance, groundedness and answer relevance. Judges are useful and they are noisy, and the noise biases the triage counts. If a faithfulness judge flags 30% of answers, with sensitivity 0.90 and a false-positive rate of 0.15 measured on a small hand-labelled sample, the corrected prevalence is:

true_rate = (observed - false_positive_rate) / (sensitivity - false_positive_rate)
          = (0.30 - 0.15) / (0.90 - 0.15) = 0.20

That is a third lower than the raw number, enough to change which stage looks worst. Calibrate every judge you use for triage against at least a hundred human-labelled examples and recompute when you change the judge model or prompt.

Sample size matters too. If 40% of 200 labelled failures are retrieval misses, the standard error is the square root of 0.4 x 0.6 / 200, about 0.035, so the 95% interval is roughly 33% to 47%. That is precise enough to choose the next project, but not to declare that a change moved retrieval misses from 40% to 37%. For before-and-after comparisons, rerun the same labelled questions through both versions and compare paired outcomes rather than two independent rates.

Online signals between evaluation runs

Labelled evaluation runs are periodic; production traffic changes daily. A few unlabelled signals, tracked per stage, catch most regressions between runs and tell you which stage to triage first:

• Retrieval: the distribution of the top candidate's score and the share of queries where it falls below a threshold that historically meant "nothing relevant". A shift after a reindex or embedding change points at retrieval before anyone files a ticket.
• Packing: the number of passages in the final prompt and the share of requests that hit the truncation path. A release that changes the template or adds conversation history shows up here immediately, while recall metrics stay flat.
• Generation: the decline rate, citation coverage (the share of answer sentences that cite a prompt passage), and the rate at which a lightweight groundedness check flags answers. Sudden moves after a generator version change belong to this stage.
• Users: thumbs-down and follow-up rephrasing ("no, I meant...") are noisy individually but useful as a trigger for sampling traces into the labelling queue.

Route a stratified sample of low-scoring or flagged traces into a weekly labelling queue rather than a uniform sample, but keep a small uniform slice too, so that the triage percentages you report are not distorted by how you chose what to label. When a signal moves, run the stage-ordered triage on a fresh sample of recent traffic before deciding on a fix. The point is not to automate the diagnosis completely, it is to make sure the first hour of an incident is spent on the stage that actually changed.

Stage-specific fixes and their side effects

Each fix has a cost and often shifts failures to the next stage, which is why triage should be rerun after every change.

• Retrieval misses. Hybrid retrieval (BM25 with dense) recovers exact names and identifiers; query rewriting recovers follow-ups; better chunking recovers answers split across chunk boundaries. Larger k is the bluntest tool: it raises recall but pushes the problem to packing.
• Ranked or packed out. A cross-encoder reranker over the k candidates moves the right passage to the top before truncation, at a cost that grows linearly with k. If the template or conversation history grew, measure the token budget left for passages; a 2,800-token system prompt on a 4,500-token budget leaves room for about three 350-token passages.
• Generation errors. Liu and colleagues showed that models use information at the start and end of a long context better than in the middle, so placing the strongest passages at the edges helps. Fewer passages also helps: every extra irrelevant passage is another chance to be distracted. A larger generator helps least per dollar unless this bucket dominates.
• Coverage gaps. No retrieval setting finds a document that does not exist. The content team owns the fix; the system's job meanwhile is to decline instead of guessing, which a strict grounding instruction does at the price of also declining some answerable questions.

A worked example: triaging the Belgavi's Book Store assistant

Suppose the hypothetical Belgavi's Book Store assistant answers questions about stock, editions and store policies, and an evaluation run over 400 labelled questions gets 55% right (illustrative figures throughout). Triage of the 180 failures gives 11 coverage gaps, 130 retrieval misses, no packing losses and 39 generation errors. The team had planned to switch to a larger generator; the triage says that change could at best fix part of the 39, and in the lab's model it costs roughly forty times more per thousand questions.

Instead they widen retrieval to 30 candidates and add a reranker. Accuracy rises to about 73%; rerunning triage shows retrieval misses down to 37, packing losses at 10, generation errors 49 (more questions now reach the model, so more can fail there). Adding BM25 to make retrieval hybrid fixes most of the remaining misses on author names and ISBNs, and accuracy reaches about 78%. Three months later a longer system prompt ships for a new feature, accuracy drops, and the first triage run after the release shows packing losses jumping while retrieval recall is unchanged: the prompt now holds only three passages. That diagnosis takes minutes because the trace logs post-truncation passage identifiers.

Common mistakes

The recurring mistakes are about measurement rather than models. Teams report only end-to-end accuracy, so every regression looks like a model problem. They log retrieved passages instead of prompt passages, so truncation is invisible. They treat a single gold passage as the only valid source and count alternative correct sources as retrieval misses. They trust an uncalibrated judge's counts. And they fix the stage with the most visible symptom, usually generation, because a wrong answer reads like a model mistake even when the model was handed the wrong evidence. A useful rule: no generator change is approved until triage shows generation errors as the largest bucket.

How the lab maps to production

The lab simulates 400 synthetic questions. Each has a chance of having no supporting document; otherwise its gold passage is scored against 150 distractors to find its first-stage rank, optionally reranked by a cross-encoder over the k candidates, kept or dropped according to the number of passages that fit the token budget, and finally answered with a probability that depends on the model, the passage's position in the context and the number of distracting passages. The funnel shows how many questions survive each stage. Mission one asks you to read the funnel and fix the stage that is really failing without upgrading the model. Mission two ships a longer prompt template that silently truncates context. Mission three updates the generator so that the same context produces more errors. Mission four adds questions the corpus cannot answer and asks you to stop the model guessing without giving up too many correct answers.

Run the lab: Triage the funnel: find which RAG stage loses the answer

The lab below follows 400 synthetic questions through a RAG pipeline. Blue bars show how many survive each stage, the coloured blocks underneath show how many were lost there and why, and the strip at the bottom splits the final outcomes into right, declined and confidently wrong. Dots animate questions moving through the funnel and falling out where they fail. Start with the naive setup, read the funnel, then load mission one.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Log the passage identifiers of the final assembled prompt, after truncation, alongside the retrieved candidates and their ranks.
2. Build or refresh a labelled set of a few hundred questions with every valid supporting passage, not just one.
3. Run stage-ordered triage over every failure and rank the buckets: coverage, retrieval, packing, generation, evaluation error.
4. Calibrate any LLM judge used for triage against at least a hundred human labels and correct its counts for false positives.
5. Measure how many passages actually fit your token budget after the system prompt and history, and alert when it shrinks.
6. Rerun triage after every pipeline change, because fixing one stage moves failures to the next.

Sources and further reading

• Seven Failure Points When Engineering a Retrieval Augmented Generation System (2024)
• RAGAS: Automated Evaluation of Retrieval Augmented Generation (2023)
• ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (NAACL 2024)
• Lost in the Middle: How Language Models Use Long Contexts (TACL 2024)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020)

Related in this series:

• Cross-Encoder Reranking Depth: How Many Candidates Are Worth the Latency
• Lost in the Middle: Ordering and Packing Retrieved Context for Long Prompts
• Grounded Citations in RAG: Verifying Every Claim Against Its Source Passage

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.