Evidence label: deterministic synthetic model. This lab runs a small JavaScript model of a retrieval-augmented generation pipeline. It creates 400 synthetic questions, decides whether each has a supporting passage, ranks that passage among 150 synthetic distractors, optionally reranks the top k, keeps as many passages as fit a token budget, and samples whether a small or large generator answers correctly, declines or answers wrongly. It counts where each failure happened. It does not call a retriever, reranker or language model; the probabilities and costs are illustrative assumptions.
Play the lab
How to play
Choose the retriever, the number of candidates, whether a reranker runs, how many passages go into the prompt, their order, the grounding instruction and the generator. The funnel updates instantly and the insight box names the largest failure bucket and the kind of fix that belongs to it. Inject a longer prompt template, a generator update or questions outside the corpus to see failures move between stages. Each mission loads a broken pipeline; the goal is always stated in production terms, and the third star goes to the cheaper or tighter configuration.
What the model assumes
3% of questions have no supporting document (15% with the coverage event). Gold passages score from a normal distribution that is lower for the 30% of hard questions, and higher with hybrid retrieval; distractors score from a standard normal. The reranker separates gold from distractors more sharply. Passages are 350 tokens, the prompt budget is 4,500 tokens with a 600-token base template, and the larger-template event adds 2,800 tokens. The generator answers correctly from context with a base probability, reduced when the gold passage sits in the middle of the context and for each additional passage. Without the gold passage it can still answer from memory with a small probability. A strict grounding instruction makes the model decline more often, including on a few answerable questions.
What it cannot tell you
Real failure mixes depend on your corpus, queries and models, and real stage outcomes are correlated in ways this independent model ignores: hard questions are often hard at every stage. The lab also treats triage labels as exact, while in production they come from human labels or calibrated judges with their own error. Use it to practise reading a funnel and to see how fixes shift failures between stages, then run the same triage on your own traces.
Read the full explanation
- RAG Failure Triage: Separating Retrieval Misses From Generation Errors (companion article)
- Cross-Encoder Reranking Depth: How Many Candidates Are Worth the Latency
- Lost in the Middle: Ordering and Packing Retrieved Context for Long Prompts
All AI in Production labs · Lab catalog
- Seven Failure Points When Engineering a Retrieval Augmented Generation System (2024)
- RAGAS: Automated Evaluation of Retrieval Augmented Generation (2023)
- ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (NAACL 2024)
- Lost in the Middle: How Language Models Use Long Contexts (TACL 2024)
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020)
Views are my own and do not represent any employer, past or present.