Evidence label: deterministic synthetic model. The lab is a JavaScript model of a paged-KV inference scheduler. It generates Poisson arrivals with random prompt and output lengths, allocates 16-token blocks from a fixed pool derived from the memory setting, admits requests optimistically, applies chunked prefill, and when blocks run out preempts the most recently admitted sequence by swapping its blocks to CPU or discarding them for later recomputation. Step times come from a simple linear cost model. Nothing runs on a GPU; every number is a calculated estimate.
Play the lab
How to play
Choose the preemption mode and move the memory utilisation, concurrency limit, admission headroom and arrival rate; toggle the FP8 KV cache to halve bytes per token. Every change re-runs two minutes of traffic with the same seed. Watch the blue area hit 100% and the red preemption ticks appear, and compare the tiles: throughput, p95 time to first token, the p99 gap between tokens, preemptions per thousand requests, the share of requests that paused over a second, recomputed tokens and KV capacity.
Missions unlock in order and reward the cheapest fix. The events make answers longer, add a 2.5 times traffic burst between 40 and 70 seconds, and slow the PCIe path by five times.
What the model assumes
A 7B FP16 model with 0.5 MiB of KV per token on a 48 GB GPU, 13.5 GB of weights and 2 GB of overhead. A decode step costs 25 ms plus 0.3 ms per running sequence plus 0.12 ms per prefill token, with at most 512 prefill tokens per step. Swapping costs 0.025 ms per token plus 0.15 ms per block in each direction (five times slower with the slow PCIe event) and is added to the step. CPU swap space is 4 GB; when it is full the model falls back to recompute. Prompts are uniform between 200 and 1,500 tokens and outputs exponential around 250 tokens, capped at 1,500.
What it cannot tell you
Real kernels are not linear, attention cost grows with context, real schedulers differ between versions (priorities, prefix caching, watermarks, asynchronous transfers), and the headroom slider stands in for effects you get through other limits. Absolute numbers will not match any deployment. Use the lab to understand when and why preemption happens and what each lever trades, then confirm on your own server with its preemption metrics and a load test that replays your real length distribution.
Read the full explanation
- KV Cache Preemption in vLLM: Swap, Recompute and Behavior Under Memory Pressure (companion article)
- PagedAttention Block Tables: How vLLM Ends KV Cache Fragmentation
- KV Cache Capacity Planning: How Many Concurrent Users Fit on One GPU
All AI in Production labs · Lab catalog
- Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023)
- vLLM documentation: engine arguments
- vLLM source code and scheduler (GitHub)
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024)
Views are my own and do not represent any employer, past or present.