Evidence label: deterministic synthetic model. This page runs a small JavaScript model of a three-tier LLM deployment. It generates 2,000 synthetic queries with hidden difficulty, decides whether each tier answers each query correctly from a logistic skill curve, and draws a noisy confidence score for every answer. A cascade or a router then decides which answer the user receives. Costs and latencies are illustrative constants. No model is called and no provider's pricing or accuracy is represented; all numbers are calculated estimates.
Play the lab
How to play
Pick a strategy: always large, a two-tier or three-tier cascade, or a router. For cascades, the thresholds set how confident the small and medium models must be for their answer to be kept. For the router, the same sliders set how likely the router must judge a cheaper model to succeed before sending the query there. The noise sliders control the quality of the confidence signal and of the router's difficulty estimate. Every change re-runs the same queries, so differences come from your settings; use a new seed to check robustness.
The four missions unlock in order and award up to three stars, mostly for a lower cost per thousand queries once the quality and latency goals are met.
What the model assumes
Sixty percent of queries are easy, thirty percent medium and ten percent hard. The chance a tier answers correctly is a logistic function of its skill minus the query's difficulty, sampled independently per tier. Confidence for a correct answer averages 0.74 minus a small difficulty term, and 0.42 for a wrong one, plus Gaussian noise; the overconfidence event raises the small model's confidence on wrong answers to 0.70. The router estimates difficulty with Gaussian error, biased low by 0.12 when traffic gets harder. Prices are $0.30, $2.00 and $8.50 per thousand queries, plus $0.02 for the router; completion latencies are 0.6, 1.5 and 3.5 seconds with 12% jitter.
What it cannot tell you
Real model errors are correlated across tiers, difficulty is multi-dimensional rather than a single number, and confidence signals behave differently by task and by prompt. Grading correctness is itself noisy in production. Treat the shapes as the lesson: confident mistakes drive the accuracy gap, escalation drives latency, and routers drift. Then measure your own models on your own labelled traffic before changing anything in production.
Read the full explanation
- Model Cascades and Routing: Cutting LLM Cost Without Losing Answer Quality (companion article)
- Semantic Caching for LLMs: Similarity Thresholds and the Wrong-Answer Risk
- TTFT, TPOT and Goodput: Setting LLM Latency SLOs That Mean Something
All AI in Production labs · Lab catalog
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (Chen, Zaharia, Zou, 2023)
- RouteLLM: Learning to Route LLMs with Preference Data (Ong et al., 2024)
- AutoMix: Automatically Mixing Language Models (Madaan et al., 2023)
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing (Ding et al., ICLR 2024)
- Language Model Cascades: Token-Level Uncertainty and Beyond (Gupta et al., ICLR 2024)
Views are my own and do not represent any employer, past or present.