Evidence label: deterministic synthetic model. This page runs a small JavaScript model of KV cache allocation on one GPU. Requests arrive at random with random prompt and output lengths. In contiguous mode a first-fit allocator reserves prompt plus max_tokens per sample in a linear address space; in paged mode a free list hands out fixed-size blocks as sequences grow, prompt blocks are shared between parallel samples, and the newest request is preempted and recomputed when the free list runs dry. Step time grows with batch size, context and admitted prompt tokens. Nothing here runs a model or measures a GPU; all numbers are calculated estimates.
Play the lab
How to play
Choose an allocator, then adjust block size, KV memory, max_tokens, the running-sequence cap and the arrival rate. Every change re-runs the same seeded traffic, so differences come from your change. The memory map animates through snapshots of the run; the dashed line on the chart marks the moment shown. Metric tiles show the change against your previous setup in green or red. Missions unlock one at a time: Load mission setup starts from the broken configuration, and stars reward efficient fixes such as less memory or a sensible block size. Toggle parallel sampling, long-report users or a traffic burst at any time.
What the model assumes
One token of KV costs 128 KiB, so 1 GB in the lab holds 8,192 tokens. Prompts are uniform between 200 and 2,000 tokens; outputs are exponential with a mean of 250 tokens (six times longer for one request in five with long-report users) and are capped by max_tokens. A step costs 9 ms plus 0.15 ms per running sequence plus a term proportional to total context, multiplied by a paging overhead that grows as blocks get smaller, plus prefill time for newly admitted or recomputed tokens. The paged allocator keeps a 1% free-block watermark at admission, like vLLM.
What it cannot tell you
Real kernels are not linear, block-size effects depend on the attention backend, and real schedulers add chunked prefill, swapping, prefix caching, priorities and multi-GPU sharding. The fragmentation you see depends on the synthetic length distribution. Use the lab for the shape of the trade-offs, then read your own server's KV usage and preemption metrics under a replay of real traffic.
Read the full explanation
- PagedAttention Block Tables: How vLLM Ends KV Cache Fragmentation (companion article)
- KV Cache Capacity Planning: How Many Concurrent Users Fit on One GPU
- Prefix Caching With Radix Trees: Reusing Prompt Work Across Requests
All AI in Production labs · Lab catalog
- Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023)
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (vLLM blog)
- vLLM documentation: engine arguments
- vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
- Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022)
Views are my own and do not represent any employer, past or present.