Query Rewriting and HyDE: Fixing the Question Before Retrieval Runs

Why the raw query misses

Dense retrieval compares the embedding of the query with the embeddings of passages. That works when the query and the passage say similar things in similar words. Three common query shapes break the assumption:

• Context-free follow-ups. In a chat, the subject lives in earlier turns. "Is it available in hardback?" embeds near every passage about hardbacks, not near the book under discussion.
• Vocabulary mismatch. Users describe what they want; documents describe what they contain. Short, vague queries land between many passages and rank none of them decisively.
• Exact identifiers. ISBNs, order numbers, error codes and product codes carry all their meaning in a string that embeddings represent poorly and lexical search represents perfectly. These queries usually work as typed and are the easiest to break by rewriting.

A fourth group, well-formed questions with the right vocabulary, needs no help. The first engineering decision is therefore not which rewriting technique to use but which queries should be rewritten at all. A rewrite step that improves vague queries by 30 points and silently breaks 5% of exact-ID lookups can still be a net loss if those lookups are the ones customers escalate.

Condensing: turning a conversation into a standalone query

The cheapest and most reliable rewrite is condensing. The model receives the last few turns and the new message and returns a single standalone question: "Is The Lighthouse Keeper's Daughter available in hardback?" The output is short, typically 15 to 40 tokens, so the call is fast, and the task is close to extraction, so small models do it well.

System: Rewrite the user's last message as a standalone search query.
Keep names, identifiers and numbers exactly as written. Do not answer.

History:
user: Do you have the new lighthouse novel by the same author?
assistant: Yes, The Lighthouse Keeper's Daughter, published in March.
user: and the paperback?

Standalone query: The Lighthouse Keeper's Daughter paperback edition availability

Two production details matter. First, condensing must preserve identifiers verbatim; instruct it explicitly and test it, because models like to normalize. Second, the condensed query should be logged next to the original, since it is the single most useful artifact when debugging a bad answer. Frameworks package this step: LangChain has a history-aware retriever helper and LlamaIndex has condense-question chat engines; check the current documentation for names and signatures, which have changed across releases.

Multi-query expansion and rank fusion

Multi-query expansion asks the model for n paraphrases of the (condensed) question, runs a retrieval for each, and merges the ranked lists, usually with reciprocal rank fusion (RRF). RRF scores a document as the sum over lists of 1 / (k + rank), with k = 60 as the common default from the original paper. The arithmetic explains both its strength and its main failure mode:

RRF(d) = sum over lists of 1 / (60 + rank_in_list(d))

document at rank 1 in one list only      : 1/61             = 0.0164
document at rank 3 in three of four lists: 3 x 1/63         = 0.0476
document at rank 8 in all four lists     : 4 x 1/68         = 0.0588

Fusion rewards consensus far more than a single strong vote. That is what you want for vague queries: the right passage tends to appear in several paraphrases' lists, while each paraphrase's noise lands on different wrong passages. It is exactly what you do not want for an exact-ID query, where the original query ranks the right passage first and three paraphrases that dropped the identifier agree on a different passage from the same topic. The consensus outvotes the one list that was right.

Cost and latency scale with n in different ways. The paraphrases come from one call whose output grows by roughly 20 tokens per variant, so generation time grows linearly. The retrievals run in parallel, so they add only a little latency, but they multiply load on the vector store. Most of the recall gain usually comes from the first two or three variants; measure where the curve flattens on your own traffic rather than defaulting to five.

HyDE: search with a hypothetical answer

Hypothetical Document Embeddings (HyDE), from Gao and colleagues, take a different route around vocabulary mismatch. Instead of paraphrasing the question, the model writes a short passage that would answer it. That passage is embedded and used as the query vector. Because it is written in the style and vocabulary of a document, it tends to land closer to real documents than the question does, even when its facts are wrong. The paper samples several hypothetical passages and averages their embeddings together with the embedding of the original query, which matters more than it looks.

The weakness is that the model writes from what it knows. For entities it has seen, the hypothetical passage is a good stand-in for the real one. For new products, internal documents or anything after its training data, it writes a fluent passage about something plausible: usually a better-known neighbour. Every sample makes the same guess, so averaging does not cancel the error, and the search lands on the neighbour's passages. This is query drift, and it is worst exactly where RAG is supposed to help most, on content the model does not already know.

HyDE is also the most expensive rewrite. A hypothetical passage is 80 to 150 output tokens per sample, against 20 for a paraphrase, and output tokens dominate both the latency and the price of the call. Query2doc, a related method, concatenates the generated passage with the query for sparse or dense retrieval instead of replacing it, which keeps the original terms in play.

The latency and cost arithmetic

A rewrite step sits on the critical path, before retrieval, so its latency adds directly to time-to-first-token. A simple model is time-to-first-token of the rewriter plus output tokens times per-token time. With illustrative numbers, not benchmarks:

Those are medians. LLM endpoints have long tails, and p95 is what users feel, so a call with a 500 ms median can easily show 1 to 2 seconds at p95. Cost follows the same shape: with an illustrative small-model price of $0.15 per million input tokens and $0.60 per million output tokens, a 450-token prompt and 135 output tokens costs about $0.00015 per query, or $0.15 per thousand; three parallel HyDE samples cost three times that. At large-model prices around twenty times higher the same rewrite costs several dollars per thousand queries, which can exceed the cost of the answer generation itself for short answers.

Always put a timeout on the rewrite call and fall back to the raw (or previously condensed) query when it fires. A fallback loses the rewrite's benefit for that query but keeps the page from hanging when the rewriter's provider has a bad hour. Mission three in the lab is built around this: the rewriter slows threefold and a timeout alone throws away most of the gain, so you have to decide which part of the rewrite is worth waiting for.

Guardrails: keep the original, route the identifiers

Three structural choices prevent most rewrite damage:

1. Keep the original query in the search. Add its result list to the RRF fusion or its embedding to the HyDE average. The original carries evidence the rewrite may have lost (identifiers, rare words, the user's own phrasing), and it bounds how far a hallucinated passage can drag the search.
2. Route exact identifiers around the rewriter. A regular expression or a small classifier that detects ISBNs, SKUs, ticket numbers and error codes, then sends those queries straight to lexical or hybrid search, removes the riskiest class of rewrites and saves the LLM call for them.
3. Constrain the rewrite. Tell the model to preserve names and numbers verbatim, cap output length, and for HyDE prefer neutral phrasing over specific facts. Then verify: a cheap post-check that every identifier in the original appears in the rewrite catches most normalization bugs.

Hybrid retrieval complements all of these. If BM25 runs on the original query alongside dense retrieval on the rewrite, exact matches survive even when the rewrite drifts.

A worked example: rewriting for Belgavi's Book Store

Suppose the hypothetical Belgavi's Book Store assistant receives 100,000 queries a day: 40% vague browsing questions, 20% follow-ups, 20% ISBN or order lookups and 20% well-formed questions. Offline evaluation without rewriting (illustrative numbers) gives hit@5 of 55% on vague queries, 15% on follow-ups and close to 100% on the other two, about 65% overall.

Condensing with a small model lifts follow-ups to roughly 85% and the overall rate to about 79%, for around $8 a day (100 thousand queries at $0.08 per thousand) and a p95 of about half a second. Adding three paraphrases with RRF lifts vague queries to about 80% and the overall rate to about 90%, for roughly another 300 to 500 ms of p95. Turning off "keep the original" or the ISBN router at this point does not move the average much but breaks a few percent of lookups that used to work, which shows up as support tickets rather than in the dashboard. Switching to a large model for HyDE might add a few points on vague queries for about $10 per thousand queries in illustrative pricing (around $1,000 a day) and a p95 near three seconds. The decision is then explicit: are a few points of hit@5 worth $1,000 a day and a slower first token? These are calculated, illustrative figures; the method, measuring per query type and counting harmed queries separately, is what transfers.

Detecting rewrite failures in production

An average hit rate hides rewrite damage, because gains on one query type cancel losses on another. Instrument the step directly:

• Log original and rewritten queries side by side, with the router decision and whether a timeout fired.
• Track a harmed-query rate offline. On a labelled set, count queries that hit without rewriting and miss with it. Treat it as a separate SLO from the overall hit rate.
• Check identifier preservation online. If the original contains a pattern that looks like an ID and the rewrite does not, increment a counter. A sudden rise usually means a prompt or model change.
• Segment by entity novelty. Report hit rate separately for items added after the rewriter model's training data. HyDE drift concentrates there.
• Watch rewrite latency percentiles and fallback rate as first-class retrieval metrics, not as part of a generic LLM dashboard.

Common mistakes are predictable: shipping HyDE because a benchmark on public data improved without testing on new items, letting a rewrite drop the original query entirely, sending every query through the most expensive model, and evaluating with average hit rate only.

How the lab maps to production

The lab builds a synthetic catalogue of 120 items with four passages each, grouped into topics with similar sibling items, and a stream of 200 queries of four types: vague, follow-up, exact ID and well-formed. A simulated rewriter condenses follow-ups (when the model keeps the subject), paraphrases toward the true intent only when it recognises the item, and writes HyDE passages that substitute a better-known sibling when it does not. Retrieval is real cosine search with RRF fusion. Mission one asks you to fix follow-ups within a latency budget. Mission two turns on new catalogue items and asks you to stop HyDE from breaking queries. Mission three slows the rewriter threefold. Mission four combines follow-ups and new items and asks for quality, latency and cost targets at the same time, where the best answer spends on the model but not on long hypothetical passages.

Run the lab: Rewrite the question: condense, multi-query and HyDE under pressure

The lab below runs a stream of synthetic queries through a rewrite step and a retriever. On the left, coloured dots are queries by type moving through the rewriter and the search; they turn green or red at the end for a hit or miss in the top 5, and a white ring marks a rewrite that timed out. On the right, grey bars show hit@5 by query type with no rewriting and coloured bars show it with your settings. Start with no rewriting, then load mission one.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Label a few hundred real queries by type (follow-up, vague, identifier, well-formed) and measure hit@5 per type with and without rewriting.
2. Add a harmed-query count to your retrieval evaluation: queries that hit before the rewrite and miss after it.
3. Route identifier-like queries (ISBNs, order numbers, error codes) around the rewriter to lexical or hybrid search.
4. Keep the original query in every fused search or HyDE average, and verify that identifiers survive the rewrite.
5. Put a timeout on the rewrite call with a fallback to the raw query, and alert on the fallback rate.
6. Compare hit@5 against rewrite cost and p95 latency for one, two and three paraphrases before choosing n.

Sources and further reading

• Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE, ACL 2023)
• Query2doc: Query Expansion with Large Language Models (EMNLP 2023)
• Query Rewriting for Retrieval-Augmented Large Language Models (EMNLP 2023)
• Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models (2023)
• Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (SIGIR 2009)

Related in this series:

• Reciprocal Rank Fusion Tuning: When BM25 Rescues Dense Retrieval Misses
• Multi-Hop Questions in RAG: Iterative Retrieval and Query Decomposition
• RAG Failure Triage: Separating Retrieval Misses From Generation Errors

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.