Evidence label: deterministic synthetic model. This page runs a small JavaScript model of two-stage retrieval. For each synthetic query it draws one to four relevant passages with grades 1 to 3, places them among the top of 4,000 non-relevant scores to form a first-stage list, and gives every candidate a cross-encoder score that depends on its true relevance plus model-dependent noise. Reranking work is pushed through a multi-GPU FIFO queue with Poisson arrivals. All data are synthetic and every number is a calculated estimate.
Play the lab
How to play
Move the rerank depth and watch nDCG@5, the share of queries with a relevant passage in the top 5, and p95 latency. Change the model size, passage length, traffic and GPU count to see how the latency curve on the chart moves while the quality curve stays put. The relevant passages inside depth tile is the ceiling: the reranker cannot exceed it.
Inject a traffic surge, longer chunks or a weaker first stage to stress a setup. Missions unlock in order; load each setup, fix it, and earn more stars by meeting the goal with less depth, fewer GPUs or higher quality. New random seed redraws the queries and arrivals.
What the model assumes
First-stage scores of relevant passages are normal with a mean that rises with grade; non-relevant scores are the top tail of 4,000 standard normals, so deep candidates are genuinely hard negatives. Cross-encoder scores weigh true relevance heavily, keep a little of the first-stage signal, and add noise that shrinks with model size. Each query's reranking costs a fixed 2.5 ms plus depth times the per-pair time, which grows with passage tokens; GPUs serve queries first come, first served, and cost an illustrative $1.20 per GPU-hour.
What it cannot tell you
Real relevance is messier than graded labels, real rerankers fail in structured ways (for example on tables or numbers), and real GPU throughput depends on batching, padding and kernels. The model ignores the LLM's own sensitivity to context order and the network hops of a remote reranking service. Use it to understand the shape of the depth decision, then benchmark your own retriever and reranker.
Read the full explanation
- Cross-Encoder Reranking Depth: How Many Candidates Are Worth the Latency (companion article)
- Reciprocal Rank Fusion Tuning: When BM25 Rescues Dense Retrieval Misses
- RAG Latency Budget: Where the Milliseconds Go Between Query and Answer
All AI in Production labs · Lab catalog
- Passage Re-ranking with BERT (Nogueira and Cho, 2019)
- MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (SIGIR 2020)
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP 2019)
- Hugging Face Text Embeddings Inference documentation
Views are my own and do not represent any employer, past or present.