Reciprocal Rank Fusion Tuning: When BM25 Rescues Dense Retrieval Misses

Why dense retrieval misses and BM25 rescues

An embedding model compresses a passage into a few hundred numbers that describe what it is about. Rare tokens such as part numbers, error codes, function names, SKUs and legal citations carry almost no semantic content the model learned to represent, so the query vector for "E4021 install" lands near the centroid of installation documents. Among fifty such documents, the one containing E4021 has no particular advantage. Domain jargon the embedding model never saw in training behaves similarly: its vector is noisier, and rankings degrade.

BM25 scores documents by the query terms they contain, weighted by inverse document frequency. A token that appears in one document out of a million has an IDF of about ln(1 + 1e6 / 1.5) ≈ 13.4, while a word in half the corpus scores near zero, so an exact rare-token match dominates the ranking. The weakness is vocabulary mismatch: "how do I get my money back" shares no terms with a page titled "refund policy", and BM25 cannot bridge it. Results from benchmark suites such as BEIR are consistent with this split: BM25 is a strong, robust baseline out of domain, dense models win in domain, and combinations are usually more robust than either alone.

In the lab's default corpus, dense retrieval reaches about 89% recall@10 on paraphrased questions but only about 34% on identifier queries; BM25 reaches 100% on identifiers and about 72% on paraphrases. Those numbers come from the synthetic model, but the shape is the one teams report when they first log per-query-type recall.

Reciprocal rank fusion from first principles

RRF, proposed by Cormack, Clarke and Büttcher in 2009, ignores scores entirely and uses ranks. Each retriever returns a ranked list, and every document receives the sum over lists of 1 / (k + rank), with ranks starting at 1 and k = 60 in the original paper. Documents missing from a list contribute nothing for it. The fused ranking sorts by that sum.

# RRF contributions, calculated
k = 60 : rank 1 -> 1/61 = 0.0164   rank 10 -> 1/70 = 0.0143   rank 40 -> 1/100 = 0.0100
k = 10 : rank 1 -> 1/11 = 0.0909   rank 10 -> 1/20 = 0.0500
k = 1  : rank 1 -> 1/2  = 0.5000   rank 10 -> 1/11 = 0.0909

# doc A: rank 1 in BM25 only        doc B: rank 5 in both lists
k = 60 : A = 0.0164                 B = 2/65 = 0.0308   -> B wins
k = 1  : A = 0.5000                 B = 2/6  = 0.3333   -> A wins

That table is the whole personality of RRF. With a large k the curve is flat: rank 1 is worth only 15% more than rank 10, so a document that both retrievers consider decent beats a document that one retriever is certain about. With a small k the curve is steep and a single list's top hit dominates. Neither is right in general. Consensus is a good signal when both retrievers are reasonable; it is a bad signal for identifier queries, where BM25 is certain and dense retrieval is guessing. Because RRF only uses ranks, it is immune to the fact that BM25 scores are unbounded and cosine similarities sit in a narrow band, which is why it works acceptably with no tuning at all.

The k constant, weights and list depth

Three knobs matter in practice. k sets how steeply rank matters, as above. Weights multiply each list's contribution, w_i / (k + rank_i), letting you trust one retriever more for a traffic mix or a query class. Depth (often called the rank window) is how many results each retriever contributes before fusion, which matters more than it looks. At k = 60, a document at rank 40 in both lists scores 2/100 = 0.020, more than first place in one list at 0.0164. Deep windows therefore let middling agreement overrule a confident single hit, and they cost more: every candidate must be fetched, deduplicated and often passed to a reranker.

In the lab, a moderate k around 10 beats both the classic 60 and the greedy 1 on this traffic, and the advantage grows when traffic shifts. With out-of-domain embeddings, k 1 drops paraphrase recall to about 76% because noisy dense top hits now dominate, while k 10 with a modest BM25 weight of 1.2 keeps overall recall near 89% and identifier recall at 100%. The general lesson: k = 60 is a safe starting point, not a tuned value, and it should be swept on your own labelled queries.

Score fusion: when it beats RRF and when it breaks

The alternative to RRF is to combine scores: normalise each list (min-max, z-score or L2) and take a weighted sum, α · bm25_norm + (1 - α) · dense_norm. Score fusion keeps information that ranks throw away: if BM25's top score is far above its second, that gap says something. In the lab, linear min-max fusion actually edges out RRF at its default k on the default traffic, because the synthetic scores are well behaved.

Its failure modes are about scale. Min-max normalisation maps every list's top result to 1.0 whether it was a strong match or the best of a bad lot, so a query where BM25 matched only stopwords still contributes a "perfect" lexical score. A single outlier compresses all other scores toward zero. The weight α that works for one query class is wrong for another, and every change of embedding model or BM25 analyser shifts the distributions. Distribution-based normalisation (Qdrant offers a distribution-based score fusion, and OpenSearch offers several normalisation techniques) reduces but does not remove this. RRF is the robust default; score fusion is worth trying once you have labelled data to tune and monitor it.

The knobs in real systems

Availability, defaults and parameter names change between releases (for example, which fusion type is the default, whether per-retriever weights are supported, and what the default window is), so check the documentation for the exact version you run. In Postgres, RRF is a few lines of SQL, which makes the knobs explicit:

WITH lex AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS r
  FROM docs, plainto_tsquery('english', :text) q
  WHERE tsv @@ q ORDER BY r LIMIT 50                 -- depth
), vec AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> :qvec) AS r
  FROM docs ORDER BY embedding <=> :qvec LIMIT 50
)
SELECT id, sum(w / (10 + r)) AS score                 -- k = 10, w per list
FROM (SELECT id, r, 1.2 AS w FROM lex UNION ALL SELECT id, r, 0.8 AS w FROM vec) u
GROUP BY id ORDER BY score DESC LIMIT 10;

Note that Postgres ts_rank_cd is not BM25; dedicated search engines or extensions provide true BM25 scoring. For RRF that difference matters less than for score fusion, because only the order is used.

Worked example: a parts catalog assistant

Consider the hypothetical Belgavi's Book Store expanding into printer and scanner accessories, with a support assistant over 50,000 manual pages. Query logs show 20% of questions contain a model or part number. With dense-only retrieval, the lab-style numbers suggest about a third of those identifier queries find the right page, so roughly 13% of all questions start from the wrong context. Hybrid retrieval with RRF at default settings brings identifier recall above 90% and overall recall from about 86% to about 95%: on 10,000 questions a day, about 900 more answers grounded in the right page. Tuning k to around 10 adds roughly another 250. The extra cost is a BM25 query per request, a few milliseconds on an inverted index, plus fusing 100 candidates. All figures are calculated from the synthetic model and stated assumptions.

The store's next surprise would be traffic shift. A product launch brings jargon the embedding model has never seen; dense rankings get noisier while BM25 stays stable. A per-query-type recall dashboard shows the drop within a day, and the fix is a weight or k change, not a new model.

Evaluating a fusion change before you ship it

Fusion settings are cheap to change and easy to get wrong, so put an offline evaluation in front of them. Start from a few hundred to a few thousand real queries, each labelled with the document or passage that answers it. Label by query class as well: identifier lookups, keyword queries, natural-language questions, and any class your product cares about such as error messages or code snippets. Then, for each candidate configuration, run both retrievers once, cache the ranked lists, and compute fusion offline. Because RRF only needs ranks, a sweep over k, weights and depth takes seconds once the lists are cached, which makes it practical to test dozens of settings.

Report recall@k and nDCG@k per class, not just the average, and look at the fused rank of the labelled document rather than only whether it appears. Moving the right passage from rank 9 to rank 2 matters if a reranker or the prompt only looks at the top five. Check the distribution of how many fused results came from each retriever: if one list contributes almost nothing, either it is not helping or its weight is too low. Finally, compare with each retriever alone. A hybrid that is worse than BM25 on identifier queries, or worse than dense retrieval on questions, usually has a tokenizer problem or a depth problem, not a fusion problem.

Online, the cheapest signals are the share of answers that cite a document found only by the lexical list, the per-class click or thumbs-up rate, and the rate of answers that say the information was not found. A sudden change in the lexical share after an embedding model upgrade is a useful early warning that dense rankings shifted.

Failure modes and how to detect them

• Identifier queries failing silently. Tag queries that contain codes or rare tokens with a regular expression and track their recall separately; averages hide them.
• Analyser mismatches. BM25 rescues nothing if the tokenizer splits "E4021" into "e" and "4021" or lowercases a case-sensitive code. Test the analyser on real identifiers.
• Deep windows drowning confident hits. Recall falls as depth grows past some point. Sweep depth; do not assume more candidates is safer.
• Junk lexical matches. Queries made of common words produce arbitrary BM25 lists that still contribute to RRF. Down-weight the lexical list when the query has no rare terms, or skip it.
• Duplicate documents. The same passage indexed twice can occupy two slots in each list and two in the fused list. Deduplicate before fusion.

Common mistakes: adopting k = 60 as gospel, fusing normalised scores without per-query-type evaluation, measuring hybrid recall only on paraphrased benchmark questions that never contain identifiers, and forgetting that fused candidates feed a reranker whose cost scales with depth.

How the lab maps to these decisions

Mission one starts from dense-only retrieval and asks you to lift identifier recall without hurting paraphrased questions, which any fusion does; three stars require tuning k. Mission two turns on out-of-domain embeddings and asks you to keep RRF but rebalance weights and k so the noisy dense list stops dominating. Mission three floods the system with identifier queries starting from k 120, where consensus drowns BM25's exact match. Mission four starts with 200 candidates per list under vocabulary mismatch and asks for higher recall with far fewer candidates, which works because shallower lists stop mediocre agreement from outranking confident hits.

Run the lab: Rank fusion lab: let BM25 rescue what embeddings miss

The lab below builds a synthetic corpus of 1,200 documents with topical words, distinctive terms and unique identifiers, indexes it for BM25 and for dense retrieval, and runs 240 queries of three kinds: identifier lookups, keyword queries and paraphrased questions. The canvas cycles through one example of each kind, showing the BM25 list, the dense list and the fused list in the middle, with the target document in green and lines showing where each fused result came from. Start with dense only, then switch to reciprocal rank fusion.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Tag production queries that contain identifiers or rare tokens, and report recall for them separately from paraphrased questions.
2. Check that your BM25 analyser keeps part numbers, error codes and function names intact.
3. Add a lexical retriever and fuse with RRF at k = 60 as a baseline, then sweep k from about 5 to 100 on labelled queries.
4. Sweep the per-retriever depth (rank window) and pick the smallest value where recall stops improving.
5. Try per-retriever weights for known query classes, and re-check them whenever you change the embedding model.
6. Deduplicate candidates before fusion and log the fused rank of the cited document for every answer.

Sources and further reading

• Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (Cormack, Clarke and Büttcher, SIGIR 2009)
• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
• The Probabilistic Relevance Framework: BM25 and Beyond (Robertson and Zaragoza)
• Elasticsearch reference: reciprocal rank fusion
• Qdrant documentation: hybrid and multi-stage queries

Related in this series:

• Cross-Encoder Reranking Depth: How Many Candidates Are Worth the Latency
• Query Rewriting and HyDE: Fixing the Question Before Retrieval Runs
• HNSW Tuning for RAG: M, efSearch and the Recall Versus Latency Trade

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.