Evidence label: deterministic synthetic model. This page runs a small JavaScript model of one GPU serving a synthetic 8B-class decoder. Requests arrive at random with random prompt and output lengths; a continuous-batching scheduler admits them when their KV blocks fit, grows their cache as they decode, and preempts the newest sequence when memory runs out. Step time is the time to read the weights and every running sequence's KV cache at the card's bandwidth, plus prefill compute. A separate quality probe runs real attention over synthetic per-head keys and values and measures how much the output changes when heads are mean-pooled into groups or rounded to FP8. Nothing here runs a real model or measures a GPU; all numbers are calculated estimates.
Play the lab
How to play
Choose the number of KV heads, the uptraining budget for a converted checkpoint, the cache precision and the GPU, then set the traffic. Every change re-runs the same seeded traffic, so differences come from your change. Metric tiles show the change from your previous setup in green or red. Toggle events to stress the deployment: a long-document day triples prompt lengths, a burst doubles arrivals for twenty seconds, and a memory squeeze takes 4 GB from the cache.
Missions unlock in order. Load mission setup loads a broken deployment; fix it, then try for three stars, which reward the higher-quality or cheaper answer.
What the model assumes
The model has 32 layers, 32 query heads of dimension 128 and 16 GB of FP16 weights. The server keeps 90% of GPU memory, minus 1.5 GB for activations, and allocates cache in 16-token blocks. Prompts are uniform between half and one and a half times the chosen mean; outputs are exponential with a mean of 256 tokens. A decode step costs 1.5 ms plus bytes read divided by bandwidth; prefill runs in 512-token chunks at 60% of the card's nominal FP16 throughput. Card prices are illustrative. The quality proxy maps the probe's attention-output error to a 0-100 scale where MHA is 100; uptraining is modelled as repairing a fraction of the pooling error, up to 90%.
What it cannot tell you
The quality proxy is not an evaluation of any real model: real quality loss depends on the architecture, the data and how the model was trained, and models trained with GQA from scratch do not pay a conversion penalty at all. The step model ignores kernel efficiency, attention compute, tensor parallelism, prefix caching and CPU overheads. Use the lab to build intuition for how bytes per token drive capacity and latency, then measure your own server and run your own evaluations.
Read the full explanation
- Grouped-Query Attention: How Sharing KV Heads Shrinks Inference Memory (companion article)
- RoPE and Long Context: Why Models Fail Past Their Trained Length
- Why INT4 Breaks Some Layers: Outliers, Group Size and Activation-Aware Scaling
All AI in Production labs · Lab catalog
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al., 2023)
- Fast Transformer Decoding: One Write-Head is All You Need (Shazeer, 2019)
- Attention Is All You Need (Vaswani et al., 2017)
- Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023)
- vLLM documentation: engine arguments
- DeepSeek-V2: multi-head latent attention as another way to shrink the KV cache
Views are my own and do not represent any employer, past or present.