KV CACHE · BYTES PER TOKEN
KV cache memory: storage versus recomputation
Size the key/value cache of a fictional 32-layer decoder from heads, precision, context and batch, and compare it with recomputing the prefix every step.
Your experiment
Four sequences of 16,384 prompt tokens each generate 2,048 tokens on a decoder with 8 key/value heads in 16-bit precision. The cache peaks at 9 of the 10 GiB available. Add a fifth sequence, switch to 32 heads or 8-bit values, and then turn the cache off to see what it saves.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Fictional decoder: 32 layers, 32 query heads, head dimension 128, with 10 GiB left for the cache after weights; these are illustrative numbers, not a named model. Cache bytes = 2 × layers × KV heads × head dimension × bytes per value × tokens × sequences. Work is counted as token positions processed, not FLOPs or time. Paging, fragmentation, activations and quantisation overheads are not modeled.
Where the bytes come from
During generation each layer keeps a key and a value vector for every past token so the next token can attend to them without recomputing them. For one token that costs 2 (K and V) × layers × key/value heads × head dimension × bytes per value. The fictional decoder has 32 layers, head dimension 128 and 32 query heads; Key/value heads chooses 32 (multi-head attention), 8 (grouped-query, four query heads share each key/value head) or 1 (multi-query). With 8 heads at 16 bits one token costs 2 × 32 × 8 × 128 × 2 = 131,072 bytes, 128 KiB. Ten GiB are left for the cache after the model weights.
The opening run: 9 of 10 GiB
Four sequences each hold a 16,384-token prompt and generate 2,048 tokens, the opening value of Generated tokens. Prefill stores 16,384 × 4 tokens, 8 GiB; the chart then rises linearly to 18,432 tokens per sequence, 9 GiB, at the last generated token. The cache could hold at most 20,480 tokens per sequence at this batch size. The table repeats the end-of-run requirement for all three head counts: 36 GiB with 32 heads (out of memory), 9 GiB with 8 and 1.13 GiB with 1.
Failure boundary: one more sequence
Set Sequences in the batch to 5: prefill fills exactly 10 GiB and the first generated token would need 10.0006 GiB, so the run is out of memory at generated token 1; the limit is now 16,384 tokens per sequence. With 15,872 prompt tokens and 5 sequences the run gets 512 tokens further and fails at token 513. Batch 8 fails during prefill. Halving the bytes per value to 8 bits halves the cache to 64 KiB per token, so batch 8 fits at 9 GiB; 32 key/value heads quadruple it and fail in prefill at batch 4, but fit at batch 1 (9 GiB).
What the cache saves
Turn off Keep a KV cache. Memory no longer grows, but every generated token now needs a forward pass over the whole prefix. Producing g tokens after a prompt of P tokens processes P + g − 1 positions per sequence with a cache, and g × P + g(g − 1)/2 without one. For the opening run that is 73,724 against 142,602,240 positions, about 1,934 times the work. The ratio grows with both prompt and generation length, which is why serving systems keep the cache and manage its memory instead.
Reading the run and its limits
Play replays the sizing, prefill, decode checkpoints at each quarter of the generation, the work comparison and the outcome over sixty inspection seconds; the chart shows the complete computed run. Next event, Restart, Speed and Inspection time control the replay, and any input change recomputes it. The decoder and its 10 GiB budget are illustrative, not a named model or accelerator. Work is counted in token positions, not FLOPs or latency; paged allocation, fragmentation, activations, prefix sharing and quantisation scales are not modeled.