Evidence label: deterministic synthetic model. This page runs a small JavaScript model of multi-hop retrieval. Each synthetic question needs one, two or three passages; every simulated search draws scores for the needed passage, a few hard negatives and fifty ordinary passages, and the passage is retrieved when it ranks inside the top k. Reads can extract the wrong bridge entity, confidence decides when a loop stops, and every search and LLM call adds latency and tokens. Nothing calls a model or a search engine; every number is a calculated estimate from a synthetic model.
Play the lab
How to play
Choose a strategy, then adjust top-k per search, the iterative step limit, the stop threshold and the share of multi-hop traffic. Each change replays the same 360 questions with the same random draws, so differences come from your settings. The trace strip cycles through sample questions and shows every serial step; the timing bar shows where that question's latency went. The tiles compare each metric with your previous setup.
Missions unlock one at a time. Load mission setup starts from the broken configuration. Stars reward the cheaper, faster or leaner fix. The ambiguous-entities and slow-LLM events can be toggled any time to test how robust your setup is.
What the model assumes
A search whose query contains the needed entity scores that passage around 2.3 standard deviations above ordinary passages; a search that lacks the entity scores it around 0.5, and a decomposed sub-question for a bridge hop around 0.9. Three hard negatives sit near 1.1. Reading a retrieved bridge passage extracts the right entity 94% of the time (78% with ambiguous entities), slightly less as k grows. A confident-looking wrong chain is possible; a stalled chain recovers with some probability on the next step. The decomposition planner is right 90% of the time and the router 90% of the time. Searches take about 45 ms plus 1.5 ms per passage, reads about 420 ms, routing 160 ms and the final answer about 650 ms, each with random variation. Costs use $3 per million input tokens and $15 per million output tokens, with 300-token passages.
What it cannot tell you
Real questions do not come labelled with a hop count, and real readers fail in more ways than picking the wrong entity: they can misread dates, merge two passages or refuse. The model ignores caching, parallel tool calls inside one step, rerankers and the quality of the final generation beyond a fixed accuracy. Use the lab to understand how recall, extraction errors, stop thresholds and serial latency interact, then measure the same quantities on traces from your own system.
Read the full explanation
- Multi-Hop Questions in RAG: Iterative Retrieval and Query Decomposition (companion article)
- Query Rewriting and HyDE: Fixing the Question Before Retrieval Runs
- RAG Failure Triage: Separating Retrieval Misses From Generation Errors
All AI in Production labs · Lab catalog
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (Yang et al., EMNLP 2018)
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (IRCoT, Trivedi et al., ACL 2023)
- Measuring and Narrowing the Compositionality Gap in Language Models (Self-Ask, Press et al., 2022)
- Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (Jeong et al., NAACL 2024)
- MuSiQue: Multihop Questions via Single-hop Question Composition (Trivedi et al., TACL 2022)
Views are my own and do not represent any employer, past or present.