Multi-Hop Questions in RAG: Iterative Retrieval and Query Decomposition

Two shapes of multi-hop question

The HotpotQA dataset popularised a useful split that still describes most production traffic. Bridge questions chain through an entity that the question does not name: find A, read A to learn B, then find B. Comparison questions name two or more entities and ask something about all of them: which of two editions is longer, which of two policies applies to a pre-order. Comparison questions need several passages, but the queries for each are knowable up front. Bridge questions are sequential: the query for hop two does not exist until hop one has been read. MuSiQue later built harder compositions of up to four hops specifically so that shortcuts (answering from one lucky passage) would not work.

The distinction matters because the fixes are different. Comparison questions are a recall problem that parallel sub-queries solve cheaply. Bridge questions are a dependency problem that only a loop can solve, and loops are serial, which means latency adds up hop by hop.

Why one search misses the second hop

Dense retrieval ranks passages by similarity between the query embedding and passage embeddings. For hop one, the question and the passage share the key entity, so the passage ranks near the top. For hop two, the passage is about an entity the question never mentions. Its similarity to the question comes only from topical overlap ("novel", "warehouse", "ships"), which hundreds of other passages share. In the lab's synthetic model, the answer passage for a hop whose entity appears in the query scores well above the crowd, while a bridge passage scores close to the crowd, and the arithmetic is unforgiving:

# illustrative, from the lab's synthetic score model (k = 5 passages per search)
P(hop-1 passage in top 5 | entity in query)      ~ 0.88
P(hop-2 passage in top 5 | entity not in query)  ~ 0.10
P(both in top 5 with one search)                 ~ 0.88 * 0.10 = 0.09

# increasing k helps only slowly, and every extra passage is a distractor
k = 20:  P(hop-2 in top 20) ~ 0.54, reader now sees 20 passages

Hybrid search with BM25 does not fix this either: lexical matching needs the bridge entity's name, which is exactly what is missing. Rerankers cannot promote a passage that was never retrieved. What fixes it is putting the bridge entity into a query, which means reading hop one first.

Three strategies and what they cost

There are three families of fixes, and a fourth that combines them.

• Query decomposition asks an LLM to split the question into sub-questions up front, retrieves for each in parallel, and answers from the union. One extra LLM call, one round of parallel searches. Excellent for comparison questions; weak for bridges, because sub-question two ("where does the newest novel by that author ship from?") still lacks the entity it depends on. LlamaIndex's sub-question query engine and many LangChain multi-query retrievers work this way (check your version's documentation for current names).
• Iterative retrieval interleaves retrieval and reasoning: search, have the model read the results and write the next query (or decide it has enough), search again. IRCoT interleaves retrieval with chain-of-thought steps; Self-Ask has the model pose explicit follow-up questions and answer them with search. Agentic RAG graphs in LangGraph implement the same loop as a state machine with a retrieve node, a grade node and a rewrite node.
• Single-shot retrieval stays the right answer for single-hop questions: one search, one generation, lowest latency and cost.
• Adaptive routing classifies the question first and sends it to the cheapest strategy that can answer it. Adaptive-RAG trains a small classifier on question complexity for exactly this purpose.

The arithmetic of compounding errors

An iterative chain succeeds only if every hop succeeds. Each hop has two failure points: the search must return the needed passage (recall r), and the reader must extract the right bridge entity from it (extraction accuracy e). For an H-hop chain without retries:

P(chain complete) = (r * e)^H

r = 0.88, e = 0.94  ->  per hop 0.83
H = 1: 0.83    H = 2: 0.68    H = 3: 0.57

# one retry per hop when the search misses (recall becomes 1 - (1 - r)^2 = 0.986)
per hop 0.93   H = 2: 0.86    H = 3: 0.80

Two lessons fall out. First, a retry budget is worth more than a bigger k: a second search with a reformulated query recovers most misses, while raising k adds distractors that lower e. Second, extraction errors are worse than retrieval misses, because a wrong bridge entity does not look like a failure. The next search runs on the wrong entity, returns confident-looking passages about it, and the final answer is fluent and wrong. That is why the stop condition matters as much as the search.

The stop condition is usually a self-assessed confidence or a "do I have enough to answer?" judgement. Set the threshold too low and chains stop after hop one with half the evidence. Set it too high and complete chains keep searching to verify, paying a serial LLM call each time, and incomplete chains wander until the step limit. In the lab, a threshold of 0.6 on 60% multi-hop traffic gives about 69% expected accuracy and 0.7 gives about 80%: the difference is entirely premature stops.

Latency and cost: every hop is serial

Iterative retrieval turns one round trip into several, and they cannot overlap because each query depends on the previous read. With illustrative numbers of 45 ms per search, about 420 ms per read-and-rewrite LLM call and about 650 ms for the final answer, a three-hop bridge costs roughly 3 × 465 + 650 ≈ 2.0 seconds before any queueing, against about 0.7 seconds for a single-shot answer. If the LLM endpoint slows down by a factor of two, the three-hop chain goes past four seconds while the single-shot path stays near 1.4.

Cost scales with how much each read re-reads. If every step passes k passages of 300 tokens to the model, a step with k = 5 is about 1,800 input tokens, and k = 12 is about 3,900. A three-step chain at k = 12 sends nearly 12,000 input tokens before the final answer even starts. Two practical controls keep this down: pass the reader only the new passages plus a compact running summary of what has been established, and keep k per step small because the loop, not the depth of each search, is what finds the later hops.

Watch the p95, not the mean. The mean is dominated by single-hop traffic; the tail is made of three-hop chains and chains that hit the step limit. A dashboard that shows latency by number of steps taken makes the tail explicable.

A worked example: Belgavi's Book Store

Suppose the hypothetical Belgavi's Book Store assistant receives 40% multi-hop questions: bridges like "is the sequel to the book I bought last month in stock?" and comparisons like "which of these two editions has the larger print?". The lab's synthetic model gives the following calculated numbers with k = 5, a step limit of 4 and a stop threshold of 0.8:

Iterative retrieval is the most accurate and the most expensive. The router cuts both cost and median latency by about a quarter, because single-hop questions skip the loop, but it gives up six points: some questions are misrouted, and single-shot search has no retry when it misses. Whether that trade is right depends on the product. For a support assistant where a wrong stock answer creates a refund, the iterative path may be worth it for every question; for a browsing helper the router is the better default. Either way, the numbers come from a synthetic model, and you should compute your own table from a labelled sample of your traffic.

Shortcuts at indexing time

Iteration is not the only way to reach the second hop. If the same bridges are crossed again and again, you can pay for them once at indexing time instead of on every query. Three techniques are common. Entity linking tags every chunk with canonical entity identifiers (author, title, edition, warehouse) so that, once hop one names an entity, hop two is a filtered lookup rather than a fuzzy search; vector stores such as Qdrant, pgvector and Elasticsearch all support metadata filters alongside similarity search. Contextual enrichment prepends a short description of the parent document to each chunk, so a chunk about a sequel also mentions the series and the author, which lets a single search match it. Graph-structured retrieval extracts entities and relations into a graph and walks edges for bridge questions; the GraphRAG family of approaches builds on this idea, at the cost of an extraction pipeline that must be kept fresh.

None of these removes the need for a loop on open-ended questions, but each one converts some bridge hops into direct lookups. The test is simple: sample the bridge entities your traces actually follow, and if a small set of relation types (author of, sequel of, ships from) covers most of them, index those relations explicitly.

Implementing the loop safely

The loop itself is short. The engineering is in the guard rails:

def answer(question, k=5, max_steps=4, stop_conf=0.8):
    notes, seen, query = [], set(), question
    for step in range(max_steps):
        hits = [h for h in search(query, k=k) if h.id not in seen]
        seen.update(h.id for h in hits)
        r = llm_read(question, notes, hits)          # returns facts, next_query, confidence
        notes.extend(r.facts)                        # keep a compact running summary
        if r.confidence >= stop_conf or not r.next_query:
            break
        if r.next_query == query:                    # no progress: reformulate once, then stop
            query = rewrite(question, notes)
            continue
        query = r.next_query
    return llm_answer(question, notes, cited=seen)

• Bound every loop with a step limit and a wall-clock deadline; log the number of steps taken per question.
• Deduplicate across steps so the reader does not re-read the same passages each round.
• Carry notes, not transcripts. Passing every previous passage forward makes each step more expensive than the last.
• Ask for the bridge entity explicitly ("which entity must be looked up next?") and log it, so you can audit wrong bridges.
• Calibrate the stop threshold on labelled chains: plot the confidence distribution for complete and incomplete chains and pick the threshold where they separate.

Failure modes and how to detect them

Most multi-hop failures are invisible in a single accuracy number, so log the trace. The ones that recur:

• Premature stop: the answer cites passages from hop one only. Detect it by comparing the number of steps taken with a classifier's estimate of hops needed, or by checking whether the final answer's entities appear in any retrieved passage.
• Wrong bridge: the chain follows a namesake or a stale entity. Ambiguous names (two authors with the same surname, two editions with the same title) are the usual cause. Detect it by checking that consecutive hops share an entity, and by sampling traces for human review.
• Runaway loops: the step-limit hit rate climbs. It usually means the stop threshold is above what the model's confidence ever reaches, or the retriever cannot find a passage that exists only in another index.
• Misrouting: the router sends a bridge question to single-shot. Track accuracy by routed path and by an offline label of question type.
• Latency regressions that come from the LLM endpoint, not retrieval. Split the trace into search time and LLM time so a slow model is not blamed on the vector database.

What the lab missions teach

Mission one starts with single-shot retrieval on traffic that is 60% multi-hop and asks for 70% accuracy without raising k above five; it is the bridge problem in its plainest form. Mission two switches on ambiguous entities and a low stop threshold, so wrong bridges stop early and confidently; you fix it by demanding more confidence and leaving room for another step. Mission three slows every LLM call by a factor of two and asks you to hold a p95 latency target, which forces you to trade verification steps against the tail. Mission four asks for good accuracy at a much lower cost with a tight p95, and the cheapest passing answer combines a router with a small k per step. Watch the trace strip: each card is a serial step, and the bar under it is the latency you are paying for.

Run the lab: Multi-hop retrieval: chain the hops without blowing the latency budget

The lab below generates 360 synthetic questions: single-hop, bridge (two or three hops) and comparison, in a mix you control. Each search is simulated against a synthetic corpus with hard negatives, and each read can extract the wrong bridge entity. The strip at the top replays one question step by step (cyan for searches, violet for LLM calls, green for a hit and red for a miss), and the bars below show expected accuracy by question type, with the share of incomplete evidence chains in red. Start with mission one.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Label 200 production questions as single-hop, bridge or comparison, and measure accuracy for each type separately.
2. Log a trace per question: queries issued, passages retrieved, bridge entity chosen, confidence at each step and steps taken.
3. Add a step limit and a wall-clock deadline to every retrieval loop, and alert on the share of questions that hit the limit.
4. Calibrate the stop threshold by plotting reader confidence for complete and incomplete chains on labelled examples.
5. Try a router that sends single-hop questions to single-shot retrieval, and compare cost, p50 and p95 latency and accuracy by type.
6. Keep k per step small and carry a running summary instead of every earlier passage.

Sources and further reading

• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (Yang et al., EMNLP 2018)
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (IRCoT, Trivedi et al., ACL 2023)
• Measuring and Narrowing the Compositionality Gap in Language Models (Self-Ask, Press et al., 2022)
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (Jeong et al., NAACL 2024)
• MuSiQue: Multihop Questions via Single-hop Question Composition (Trivedi et al., TACL 2022)

Related in this series:

• Query Rewriting and HyDE: Fixing the Question Before Retrieval Runs
• RAG Failure Triage: Separating Retrieval Misses From Generation Errors
• RAG Latency Budget: Where the Milliseconds Go Between Query and Answer

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.