Evidence label: deterministic synthetic model. This page runs a small JavaScript model of an LLM server: requests arrive at random, each has a random prompt and output length, and a scheduler forms batches under one of two policies. Step time grows linearly with batch size and prompt tokens. Nothing here runs a model, contacts a server or measures a GPU; all numbers are calculated estimates that make the scheduling mechanism visible.
Play the lab
How to play
Pick a scheduler, then adjust the batch size, arrival rate and mean output length. Every change re-runs sixty seconds of synthetic traffic with the same random seed, so differences you see come from your change, not from luck. The metric tiles show the change against your previous setup in green (better) or red (worse). Use New random seed to check that a conclusion survives different traffic.
The missions unlock one at a time. Use Load mission setup to start from the broken configuration, then fix it. Each mission awards up to three stars: one for meeting the goal, more for meeting it efficiently, for example with fewer batch slots. Inject the burst or long-output event at any time to stress your setup.
What the model assumes
Arrivals follow a Poisson process at the chosen rate (2.5 times higher between 20 and 40 seconds during a burst). Prompt lengths are uniform between 200 and 2,000 tokens, and output lengths are exponentially distributed around the chosen mean; with the long-output event, one request in ten generates six times more. A decode step costs 8 ms plus 0.3 ms per sequence in the batch plus 0.004 ms per newly admitted prompt token. Static batching waits until the batch is full or the fill timeout expires, then runs until every member finishes. Continuous batching admits from the queue before every step.
What it cannot tell you
The step model is linear and ignores attention cost growing with context length, KV cache memory limits, preemption, chunked prefill, multiple GPUs and network streaming overhead. Real servers have all of these, so absolute numbers here will not match any deployment. Use the lab to build intuition about the shape of the trade-offs, then measure your own server with an open-loop load test that replays your real length distribution.
Read the full explanation
- Continuous Batching Explained: Why Static Batches Leave GPUs Idle (companion article)
- PagedAttention Block Tables: How vLLM Ends KV Cache Fragmentation
- Chunked Prefill: Stopping Long Prompts From Freezing Every Other Stream
All AI in Production labs · Lab catalog
- Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022)
- Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023)
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024)
- vLLM documentation: engine arguments
- Hugging Face Text Generation Inference: launcher arguments
Views are my own and do not represent any employer, past or present.