Evidence label: deterministic synthetic model. This page runs a small JavaScript model of a GPU fleet serving LLM requests. Requests arrive at random with exponentially distributed prompt and output lengths. In colocated mode each GPU runs continuous batching and prefills newly admitted prompts inside its decode steps; in disaggregated mode prefill GPUs batch prompts up to 8,192 tokens, each request's KV cache is transferred over a serialised link, and decode GPUs run pure decode steps. Each GPU advances on its own clock. Nothing here runs a model or measures hardware; all numbers are calculated estimates.
Play the lab
How to play
Choose the serving layout, the number of GPUs, how many of them prefill when disaggregated, the KV link bandwidth, the mean prompt length and the request rate. Every change re-runs the same seeded traffic, so differences come from your change. Inject a long-document surge, a degraded link or a traffic burst at any time. Missions unlock one at a time; Load mission setup starts from the broken configuration, and stars reward smooth streams or fewer GPUs.
What the model assumes
Prefill costs 0.05 ms per prompt token plus 6 ms per batch. A decode step costs 8 ms plus 0.25 ms per sequence plus a small term per context token, with at most 64 sequences per GPU. KV is 0.125 MB per token, transferred after prefill at the link bandwidth plus 2 ms, one transfer at a time per prefill GPU. Requests go to the least-loaded GPU of the right pool. A request is good if its first token arrives within 1.5 s (counted when the decode side emits it) and its average time per output token is at most 25 ms. Mean output is 220 tokens.
What it cannot tell you
The model ignores KV memory limits on decode GPUs, layer-by-layer overlap of transfer with prefill, tensor parallelism, chunked prefill, network contention between flows and the cost of reconfiguring pools. Real crossover points depend on your model, hardware and traffic. Use the lab to understand the shape of the trade-offs, then measure goodput per GPU on your own stack.
Read the full explanation
- Disaggregated Prefill and Decode: When Splitting the GPU Pool Pays Off (companion article)
- Chunked Prefill: Stopping Long Prompts From Freezing Every Other Stream
- TTFT, TPOT and Goodput: Setting LLM Latency SLOs That Mean Something
All AI in Production labs · Lab catalog
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI 2024)
- Splitwise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024)
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024)
- NVIDIA Dynamo (GitHub)
Views are my own and do not represent any employer, past or present.