Evidence label: deterministic synthetic model. This page runs a small JavaScript model of a RAG request pipeline. For each of 1,000 synthetic requests it draws log-normal latencies for query rewriting, embedding, vector search, keyword search, batched reranking and LLM prefill, combines them according to your settings (serial or parallel retrieval, hedging, local or hosted embedding), and estimates whether the needed passage reaches the prompt. Nothing calls a real service; every number is a calculated estimate from a synthetic model.
Play the lab
How to play
Toggle the query rewrite and hedging, pick the embedding location and retrieval mode, and set rerank depth and the number of chunks in the prompt. Every change replays the same 1,000 requests with the same random draws, so differences come from your settings. The tiles show p50 and p95 time to first token, the p95 of the part before the LLM starts, expected quality, cost and the stage with the highest p95.
Missions unlock one at a time. Load mission setup starts from the broken configuration; stars reward the faster, cheaper or higher-quality fix. Inject the embedding tail, GPU contention or long-chunk events at any time.
What the model assumes
Stage latencies are log-normal around illustrative medians: rewrite 280 ms, hosted embedding 35 ms (local 7 ms), vector search 12 ms, BM25 18 ms, rerank 22 ms per batch of 32 candidates, and LLM first-token overhead 110 ms plus prompt tokens at 12,000 tokens per second. The prompt holds 400 tokens of instructions plus 300 tokens per chunk (650 with long chunks). During the embedding incident 8% of calls take an extra 350 to 750 ms; hedging sends a second call at 70 ms and keeps the faster. Thirty percent of queries are vague: they rank the needed passage lower, rewriting helps them, hybrid search takes the better of two rankings, and reranking moves the passage near the top if it is within the candidate depth. Cost uses $3 per million input and $15 per million output tokens with 250 output tokens.
What it cannot tell you
Real stage latencies are correlated (a busy cluster slows several stages at once), queues form under load, and the LLM's prefill speed depends on batching with other requests. The quality model is a simple rank-based proxy, not an evaluation of answers. Use the lab to build intuition for how serial stages, rare slow dependencies, batch staircases and prompt size shape the latency distribution, then confirm with traces from your own pipeline.
Read the full explanation
- RAG Latency Budget: Where the Milliseconds Go Between Query and Answer (companion article)
- Cross-Encoder Reranking Depth: How Many Candidates Are Worth the Latency
- HNSW Tuning for RAG: M, efSearch and the Recall Versus Latency Trade
All AI in Production labs · Lab catalog
- The Tail at Scale (Dean and Barroso, Communications of the ACM, 2013)
- Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs (Malkov and Yashunin)
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., NeurIPS 2020)
- OpenTelemetry semantic conventions for generative AI
- Elasticsearch reference: reciprocal rank fusion
Views are my own and do not represent any employer, past or present.