Sending every request to the most capable model is the simplest way to protect answer quality and the fastest way to grow an inference bill. In most production traffic a large share of queries are routine: a lookup, a reformatting, a short factual answer. A model that costs a few percent as much would answer many of them correctly. The hard part is knowing which ones, before a user sees a wrong answer.
Two families of techniques address this. A cascade tries a cheap model first and escalates when a confidence signal says the answer is unreliable. A router predicts, before any generation, which model a query needs and sends it there once. This guide works through the cost and accuracy arithmetic of both, how to choose and calibrate thresholds, the latency each one adds, and the production failures that turn a cost saving into a quality incident. The embedded lab routes synthetic queries across three model tiers so you can see confident mistakes, drift and latency as they happen.
The economics: why a cheap first try pays
Take three illustrative tiers priced per thousand queries (about 800 input and 300 output tokens each): a small model at $0.30, a medium model at $2.00 and a large model at $8.50. These are round numbers for the arithmetic, not any provider's price list; check current pricing for the models you use.
A two-tier cascade that always calls the small model and escalates a fraction e of queries to the large model costs 0.30 + e × 8.50 per thousand. It beats large-only whenever e < 1 − 0.30/8.50, that is, whenever the small model keeps more than about 3.5% of queries. The break-even is low because the first try is so cheap. With three tiers the expected cost is:
cost = c_small + e1 × (c_medium + e2 × c_large)
# illustrative: small keeps 50%, medium keeps 60% of what reaches it
cost = 0.30 + 0.50 × (2.00 + 0.40 × 8.50) = 0.30 + 0.50 × 5.40 = $3.00 per 1k
# versus $8.50 per 1k for large-only: about 65% cheaper, before counting qualityThe savings are real only if quality holds, and quality depends almost entirely on one thing: how often the cascade keeps an answer that is wrong. Published work reports large savings on benchmark tasks. FrugalGPT (Chen, Zaharia and Zou, 2023) learned a scoring function and cascade over several commercial APIs, and RouteLLM (Ong et al., 2024) trained routers on human preference data. Treat their numbers as evidence that the approach works on those tasks, not as a forecast for your traffic.
Confident mistakes decide the accuracy gap
A cascade's accuracy loss relative to always using the large model comes from queries where the small model was wrong, the confidence score said keep, and the large model would have been right. Think of the keep decision as a classifier over the small model's answers:
# illustrative: small model correct on 60% of queries
# scorer keeps 85% of correct answers (true accept) and 10% of wrong ones (false accept)
kept = 0.60 × 0.85 + 0.40 × 0.10 = 0.55 # 55% of traffic stops at small
confident_wrong = 0.40 × 0.10 = 0.04 # 4% of traffic gets a kept mistake
# if the large model would have answered ~90% of those correctly:
accuracy_gap ≈ 0.04 × 0.90 ≈ 3.6 points
# to keep the gap under 0.5 points, the false-accept rate must fall below ~1.4%This is why a threshold that looks conservative can still leak quality. What matters is the false-accept rate at the operating point, not the scorer's average accuracy. Raising the threshold cuts false accepts but also rejects more correct answers, which raises cost. The curve between the two is set by how well the confidence signal separates right from wrong answers, so improving the signal is usually worth more than tuning the threshold.
Common confidence signals, from cheapest to most expensive:
- Token log probabilities from the small model (mean or minimum over the answer). Free if your server returns them; vLLM and other OpenAI-compatible servers accept a
logprobsrequest option. Sensitive to answer length and phrasing. - Verbalised confidence (asking the model to rate itself). Cheap but often poorly calibrated, and it shifts after fine-tuning.
- Self-consistency: sample several answers and measure agreement. Stronger, but multiplies small-model cost.
- A trained verifier or scorer, as in FrugalGPT, or self-verification as in AutoMix. Best separation, but needs labelled data and its own maintenance.
- Task checks: the answer parses as valid JSON, the SQL executes, the cited passage exists. Narrow but very precise where they apply.
Routers: decide once, before generating
A router predicts which model a query needs from the query alone. It can be a small classifier, an embedding similarity to labelled examples, or a matrix-factorisation or BERT-style model as in RouteLLM. The Hybrid LLM work (Ding et al., 2024) framed it as predicting the quality gap between a small and a large model for each query and routing with a tunable threshold.
| Property | Cascade | Router |
|---|---|---|
| Information used | The actual answer and its confidence | The query only |
| Cost of a hard query | Every tier tried, plus scoring | One model plus router cost |
| Latency of a hard query | Sum of tiers tried | Router plus one model |
| Streaming | Cannot stream the small answer until it is accepted | Streams immediately |
| Behaviour under drift | Escalates more: cost rises, quality mostly holds | Mis-routes silently: quality falls, cost may even drop |
| Accuracy ceiling | Close to large-only with a good scorer | Limited by how predictable difficulty is from the prompt |
The accuracy-ceiling row deserves emphasis. Even a perfect difficulty estimate cannot tell which particular easy-looking query the small model will fumble, because model errors are partly random. A cascade observes the outcome; a router only observes the prior. In the lab, a router with a small estimation error still trails large-only by about two points at a cost similar to a cascade that trails by a fraction of a point. The router wins on latency, which is why many systems combine them: route clearly hard queries straight to the large model and cascade the rest.
Calibrating thresholds and keeping them calibrated
Thresholds are chosen on a labelled sample of real traffic, not on a public benchmark. The procedure is the same for cascades and routers:
def pick_threshold(samples, max_gap=0.005):
# samples: list of (small_conf, small_correct, large_correct) from held-out traffic
best = None
for t in sorted({round(s[0], 3) for s in samples}):
kept = [s for s in samples if s[0] >= t]
lost = sum(1 for s in kept if s[2] and not s[1]) # confident mistakes large would fix
gained = sum(1 for s in kept if s[1] and not s[2]) # small right where large is wrong
gap = (lost - gained) / len(samples)
if gap <= max_gap:
share = len(kept) / len(samples)
if best is None or share > best[1]:
best = (t, share, gap)
return best # (threshold, share kept at small tier, accuracy gap)Labelling needs a definition of correct: exact match for structured tasks, a rubric with human or model graders for open-ended ones. If you use a large model as the judge, check its agreement with humans on a sample first, and do not let the judge be the same model as the large tier or you will measure agreement rather than correctness. Re-run the calibration whenever any model in the chain changes, including provider-side updates behind a stable model name, and monitor the escalation rate daily: a sudden drop in escalations with no change in traffic is a warning sign, not a saving.
Building it in a real serving stack
A cascade is usually implemented in the application or gateway layer rather than inside an inference server, because it spans several models that may run on different hardware or come from different providers. The moving parts are the same everywhere:
- Model endpoints per tier. Self-hosted tiers can share a cluster: a small model on a single GPU behind vLLM, SGLang or TGI, and a large model with tensor parallelism on its own node pool. Each tier needs its own capacity planning, because escalation rate decides how much traffic the large tier sees. A cascade that keeps 50% at the small tier still sends half of all traffic to the large model, and a drop to 30% kept means 40% more large-model load overnight.
- A scoring step. If the score comes from log probabilities, request them from the server (the OpenAI-compatible
logprobsoption in vLLM and similar servers; check your version for limits on how many are returned). If it comes from a verifier, deploy that verifier as its own small endpoint with its own latency budget. - A policy store. Thresholds, tier order and per-task overrides belong in versioned configuration, not code, so a recalibration can roll out and roll back like any other change. Different tasks need different thresholds: a returns-policy answer and a book recommendation do not tolerate the same error rate.
- Logging for evaluation. Record, per request, every tier tried, every score, the final tier, latency and token counts. Without this you cannot recalibrate, explain a cost change or answer which tier produced a complaint.
Prompt caching interacts with all of this. A large model whose provider or server caches a long shared system prompt may be cheaper per query than the list price suggests, and a small model that runs on your own GPUs has a marginal cost that depends on utilisation rather than tokens. Recompute the break-even with your effective costs, not list prices, before deciding how aggressive the first tier should be.
A worked example: Belgavi's Book Store
Suppose the hypothetical Belgavi's Book Store runs an assistant for order status, returns policy and book recommendations, with 2,000 labelled queries sampled from its logs. Large-only scores 94.3% on that sample at $8.50 per 1k (all figures here come from the lab's synthetic model and are illustrative). A three-tier cascade with a small-model threshold of 0.70 and a medium threshold of 0.60 keeps about 32% of queries at the small tier and reaches 93.95% accuracy at about $3.86 per 1k: a 0.35-point gap for 55% lower cost. Dropping the small threshold to 0.60 saves another $0.90 per 1k but widens the gap to 2.4 points, because confident mistakes rise faster than savings.
A month later the small model is replaced with a newer version. Its accuracy is similar, but its confidence scores are higher on wrong answers. Without recalibration, the same thresholds now keep many wrong answers and accuracy drops by more than 15 points in the lab. The fix is either to recalibrate with a better scorer or to bypass the small tier until it is recalibrated. Later still, catalogue questions get harder after a product launch. A router trained on last quarter's traffic underestimates them and loses about seven points; the cascade simply escalates more, costs $4.93 per 1k instead of $3.86, and holds quality.
Production failure modes and how to detect them
- Silent quality regression after a model update. Detect with a daily sample graded against the large model or human labels, broken down by tier that answered.
- Distribution shift. Track the router's predicted-difficulty histogram and the cascade's escalation rate; both move when traffic changes.
- Escalation storms. If the small tier fails broadly (an outage, a bad deploy), every query pays twice. Alert on escalation rate and on cost per 1k, and use a circuit breaker that routes directly to the large tier.
- Feedback blind spots. Queries kept at the small tier never get a large-model answer to compare against. Shadow-send a small random fraction to the large model for evaluation.
- Prompt and format drift between tiers. Different models need different prompts; a prompt tuned for the large model can quietly hurt the small one. Version prompts per tier.
- Safety asymmetry. Smaller models may follow refusal and policy instructions less reliably. Keep safety filters in front of every tier, not just the large one.
Common mistakes
Measuring the cascade on a public benchmark instead of your own traffic is the most frequent error; the difficulty mix decides everything. The second is choosing a threshold from the scorer's average accuracy rather than its false-accept rate at the operating point. The third is ignoring latency: a cascade that halves cost and doubles p95 time to first token is a product change, not a cost optimisation. The fourth is forgetting that a router is a model with its own training data and drift. Finally, gateway and SDK features for fallbacks (retry on error with a different model) are not cascades; they react to failures, not to answer quality.
What the lab missions practise
The lab generates 2,000 synthetic queries with a hidden difficulty and simulates whether each tier answers correctly and how confident the small model is. Mission one starts from large-only and asks you to cut cost below $4.50 per 1k within half a point of its accuracy. Mission two injects an overconfident small-model update and asks you to recover quality on a budget. Mission three makes traffic harder so that a router trained on old data mis-routes, and asks you to restore quality. Mission four sets a latency budget that the three-tier cascade cannot meet, so you have to trade a small accuracy gap for speed with a router.
Run the lab: Cascades and routers: send each query to the cheapest model that gets it right
Each dot below is one synthetic query: its horizontal position is the hidden difficulty, and its height is the small model's confidence (or, for the router, its estimate that the small model will succeed). The colour shows which tier answered, and red marks a wrong final answer. Start from large-only, switch to a cascade and move the threshold line.
Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.
What to do next
- Sample and label a few hundred to a few thousand real queries, with large-model and small-model answers graded for correctness.
- Measure what share of those queries the small model already answers correctly; if it is under about 20%, a cascade is unlikely to pay.
- Compare confidence signals (log probabilities, a verifier, task checks) by their false-accept rate at the threshold you would use.
- Pick thresholds from a maximum acceptable accuracy gap, and record expected cost and p95 latency at that point.
- Shadow-send a small random fraction of kept queries to the large model and track the disagreement rate per tier.
- Alert on escalation rate and cost per 1k, and recalibrate whenever any model or prompt in the chain changes.
Sources and further reading
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (Chen, Zaharia, Zou, 2023)
- RouteLLM: Learning to Route LLMs with Preference Data (Ong et al., 2024)
- AutoMix: Automatically Mixing Language Models (Madaan et al., 2023)
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing (Ding et al., ICLR 2024)
- Language Model Cascades: Token-Level Uncertainty and Beyond (Gupta et al., ICLR 2024)
Related in this series:
- Semantic Caching for LLMs: Similarity Thresholds and the Wrong-Answer Risk
- TTFT, TPOT and Goodput: Setting LLM Latency SLOs That Mean Something
- Logprobs and Calibration: Measuring When a Language Model Is Guessing
Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.
Views are my own and do not represent any employer, past or present.