Evidence label: deterministic synthetic model. This page runs a small JavaScript model of LLM replicas with iteration-level batching and chunked prefill. Synthetic requests arrive as a Poisson process, each with a random prompt and output length, and are routed round-robin. Each step costs the larger of a memory-bound decode time and a compute-bound prefill time. The model computes every request's TTFT and TPOT and checks them against your SLOs. It does not run a model or measure a GPU; every number is a calculated estimate chosen to make the mechanism visible.
Play the lab
How to play
Set the arrival rate and replicas, then tune the two engine knobs: maximum concurrent sequences per replica and the per-step token budget (prefill chunk). The SLO sliders define the box; missions will not count a setup that loosens them. Each change re-runs the same minute of traffic with the same seed, so differences come from your change. Watch the three bars: whichever is shorter tells you which SLO is binding. Use New random seed to check that a fix is not luck; the missions are calibrated on the default seed, and some seeds are harder.
Missions unlock in order. Each loads a broken setup and awards up to three stars, for higher attainment or for meeting the goal with fewer replicas.
What the model assumes
Prompts are uniform between 300 and 1,700 tokens; with the RAG event, a quarter of requests carry 8,000 to 16,000 tokens. Outputs are exponential with a mean of 220 tokens (minimum 16). A decode step costs 6 ms plus 0.25 ms per decoding sequence; a step that carries prefill costs at least 2 ms plus 0.02 ms per prompt token, and the step takes whichever is larger. Decode tokens count against the per-step budget, admission is first come first served, and there is no KV cache limit or preemption. A GPU-hour is priced at an illustrative $4. Arrivals stop after 60 seconds; requests unfinished after a further 60 seconds count as misses.
What it cannot tell you
Real kernels are not linear: attention cost grows with context length, prefill and decode do not overlap perfectly, and memory limits force preemption that this model ignores. Routing here is round-robin rather than least-loaded or cache-aware. Network, gateway and tokenizer time are absent. Treat the absolute numbers as illustrative and the shapes as the lesson, then measure your own system with open-loop load tests that replay your real prompt and output length distribution.
Read the full explanation
- TTFT, TPOT and Goodput: Setting LLM Latency SLOs That Mean Something (companion article)
- Continuous Batching Explained: Why Static Batches Leave GPUs Idle
- Chunked Prefill: Stopping Long Prompts From Freezing Every Other Stream
All AI in Production labs · Lab catalog
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI 2024)
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024)
- Splitwise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024)
- Google SRE book: Service Level Objectives
- vLLM documentation: engine arguments
Views are my own and do not represent any employer, past or present.