Evidence label: deterministic synthetic model. This page runs a small JavaScript model of one GPU serving LLM requests with continuous batching. Requests arrive at random with exponential prompt and output lengths, optionally with long RAG prompts or a single 96,000-token prompt. Each step's time is the larger of a memory term and a compute term proportional to the tokens in the step, plus attention for prefill tokens and fixed overhead. With chunking off, prompts are prefilled in prefill-only steps that pause decodes; with it on, decodes go first and prompt chunks fill the remaining budget. Nothing here runs a model or measures a GPU; all numbers are calculated estimates.
Play the lab
How to play
Turn chunking on or off, choose the token budget per step, cap running sequences and set the arrival rate. Every change re-runs the same seeded traffic. Toggle long-prompt RAG traffic, a single 96k-token prompt at 20 seconds, or a 2× burst. The canvas scrolls through the run; when chunking is off, look for the long red steps where every stream pauses. Missions unlock in order, start from a broken setup with Load mission setup, and award extra stars for lower TTFT, shorter stalls or more traffic served.
What the model assumes
A step costs 1.5 ms of overhead plus the larger of 7 ms (plus a small term for KV read) and 0.035 ms per token in the step, plus 2×10⁻⁷ ms per query-key pair of prefill attention. That puts the compute knee at about 200 tokens per step. Prompts average about 1,300 tokens (capped at 8,000) and, with long-prompt traffic, 12% are 16k to 32k tokens. Outputs average 200 tokens. Requests are good when TTFT is at most 2 s and their average TPOT at most 30 ms. Without chunking, waiting prompts are prefilled together up to the budget, and at least one whole prompt per step.
What it cannot tell you
Real step times are not a clean roofline, attention kernels differ, and real schedulers add KV memory limits, preemption, priorities, prefix caching and limits on concurrent partial prefills. Your knee and best budget depend on your GPU, model and precision. Use the lab to learn the shape of the trade-off, then measure step time and inter-token gaps on your own server.
Read the full explanation
- Chunked Prefill: Stopping Long Prompts From Freezing Every Other Stream (companion article)
- Continuous Batching Explained: Why Static Batches Leave GPUs Idle
- Disaggregated Prefill and Decode: When Splitting the GPU Pool Pays Off
All AI in Production labs · Lab catalog
- SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024)
- DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
- vLLM documentation: engine arguments
- Hugging Face Text Generation Inference: launcher arguments
Views are my own and do not represent any employer, past or present.