DECODE · INSPECT · EXPLAIN
KV cache growth: count every stored position
Follow context growth and reconcile cache bytes with model dimensions.
Your experiment
Double the layer count, batch or bytes per value one at a time. The persistent cache must double at the same inspected position. Turn caching off: persistent storage becomes zero, but working K/V does not.
Press Play for a full minute at 1×, or use 0.5× for two minutes. Pause to read a stage. Next stage and the timeline reveal the same computed history; changing any model input resets the inspection without autoplay.
What the calculation means
Persistent KV bytes = 2 × layers × KV heads × head dimension × batch × processed positions × bytes per value. This excludes weights, attention workspaces, allocator overhead and reserved capacity.
The timeline is teaching time, not measured inference latency. A five-stage inspection is allocated to each generated token, with the complete trace spread over 60 seconds.
Visible arithmetic
| Token | Logit | Penalized | Scaled | Softmax | Filtered |
|---|
Context and cache inspector
Offline teaching fixture, not a trained transformer. An explicit eight-token score table uses only the last context token; unknown last tokens use the “the” row. Prompt tokens are whitespace-separated, not BPE tokens. Greedy or fixed-seed sampling operates on exact visible probabilities. EOS or the selected token limit ends decoding. Cache dimensions are accounting inputs only: no attention matrices or real model weights are executed. Changing cache settings must not change this fixture’s output tokens. Cache mechanics · Decoding reference · Official AICassindra articles.
Count stored keys and values explicitly
This fixture traces autoregressive decoding and counts the payload bytes of dense keys and values. It runs an explicit eight-token score table, not attention or a pretrained language model. Storage per processed position equals two times Layers times KV heads times Head dimension times Batch size times Bytes per K/V value. The leading two represents separate keys and values. The display divides bytes by 1048576 for MiB. Allocation overhead, model weights, activations, attention scores, fragmentation, alignment, sliding windows and cache quantization metadata are excluded.
Reproduce a cache snapshot and its one-token lag
Set prompt on, greedy strategy, penalty one, maximum new tokens eight, and retain K/V enabled. Set twelve layers, four KV heads, head dimension sixty-four, batch size one and two bytes per value. Each processed position contributes 12288 bytes. The continuation is mat, period, then end-of-sequence. At the final inspection three input positions have been processed and the persistent cache contains 36864 bytes. The visible sequence has four tokens because the newly selected end-of-sequence token has not been passed through another forward call. The same one-token lag applies at every append stage. Read Context and cache inspector to distinguish visible tokens from processed positions.
Vary architecture and retention independently
Layers accepts one through 128 and scales storage linearly. KV heads accepts one through 128; this is the number of stored key/value heads, not necessarily query heads in grouped-query attention. Head dimension accepts sixteen through 256 in steps of sixteen. Batch size accepts one through sixteen equal-length sequences; the model multiplies storage without simulating distinct batched prompts. Bytes per K/V value accepts one through four as a payload assumption, without emulating a numeric datatype. Doubling any single factor doubles both working and persistent payload bytes. Uncheck Retain K/V between decode steps: persistent bytes become zero while working bytes remain necessary for the current context. Previous positions are recomputed each step. For the three-step example, cached processing handles one new position per step, while uncached processing handles one, then two, then three. These are logical position counts, not FLOPs or measured time.
Control the generated trace
Prompt fixture uses whitespace tokens and only its last token chooses the next score row. Unknown last tokens use the the row. Strategy selects greedy argmax or seeded sampling. Temperature controls score separation before softmax. Repetition penalty divides positive seen-token scores and multiplies negative ones, once per present token. Top-k candidates truncates the ranked vocabulary; Nucleus top-p then keeps a probability prefix within the renormalized Top-k set. Sampling seed selects a reproducible draw without changing probabilities. Maximum new tokens limits the continuation to one through sixteen, with end-of-sequence allowed to stop sooner. These decoding controls can change trace length and therefore the final storage count. Architecture and retention controls must never change this fixture output sequence.
Inspect playback and failure boundaries
Play distributes five stages per generated token over sixty teaching seconds. Pause freezes the result. Next stage advances to the next stage in that trace; its time increment depends on the number of generated tokens. Inspection seeks directly from zero to sixty, and Restart returns to zero without changing parameters or speed. Speed selects quarter, half, normal or double progression. The double-speed run lasts thirty seconds; none of these durations measures inference latency. Any parameter edit pauses and restarts inspection. Empty prompts and more than thirty-two whitespace tokens clear stale results and disable playback. The input field caps text at 256 characters. The visible arithmetic table supplies exact rounded scores and probabilities, and the context inspector exposes integer byte totals. Use these textual outputs on a narrow phone where the canvas labels are small. This payload model cannot predict throughput, memory allocator peaks or the performance benefit of a real cache implementation.