Evidence label: deterministic synthetic model. This page runs a small JavaScript model of KV cache capacity planning. It derives each GPU's KV pool from memory size, the utilisation fraction, model weights, a fixed 2 GB reserve, the model's KV shape, the KV dtype and tensor parallelism, then simulates one replica serving its share of synthetic traffic with 16-token blocks, a 512-token prefill chunk per step and recompute preemption. Step time is a roofline over weight and KV reads, compute and a tensor-parallel communication term. Nothing here runs a model or measures a GPU; all numbers are calculated estimates.
Play the lab
How to play
Pick a model and GPU class, set the memory fraction, KV dtype, tensor parallel degree and replica count, then the fleet's request rate and mean output length. Every change re-runs the same seeded traffic. Toggle a long-context launch or a traffic burst. If the weights do not fit, the canvas says so and the server does not start. Missions unlock in order; stars reward fewer GPUs and lower GPU-hours per million output tokens.
What the model assumes
Weights are 2 bytes per parameter. Each GPU reserves 2 GB for activations and overhead. Prompts are about 1,800 tokens on average (capped at 8,000), or 16k to 32k for a quarter of requests during a long-context launch; outputs are exponential around the chosen mean. Traffic splits evenly across replicas. A decode step streams the weight shard and the active KV at the GPU's bandwidth or is limited by compute, whichever is slower, plus overhead and a per-layer communication cost when TP is above one. GPU classes are generic: 24, 48, 80 and 141 GB with illustrative bandwidth and throughput.
What it cannot tell you
Real reserves depend on your server, batch limits and graph capture; real kernels are not a clean roofline; FP8 quality effects are not modelled; and real fleets add routing imbalance, prefix caching and autoscaling. Use the lab to reason about which constraint binds, then confirm with your server's own KV log line and a load test on your traffic.
Read the full explanation
- KV Cache Capacity Planning: How Many Concurrent Users Fit on One GPU (companion article)
- PagedAttention Block Tables: How vLLM Ends KV Cache Fragmentation
- Grouped-Query Attention: How Sharing KV Heads Shrinks Inference Memory
All AI in Production labs · Lab catalog
- Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023)
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (multi-head latent attention)
- vLLM documentation: engine arguments
- Hugging Face Text Generation Inference: launcher arguments
Views are my own and do not represent any employer, past or present.