The KV cache is the single largest piece of runtime memory in LLM inference that is not the model weights — and it is pure engineering, not a part of the model. It exists to delete a staggering amount of repeated arithmetic: without it, generating each new token would force the model to recompute the keys and values of every token before it, turning linear-time generation into quadratic waste. Cache those tensors once and decode becomes cheap in compute but expensive in memory — and that trade is the whole story of modern serving. This piece derives the cache from first principles: why recomputation happens without it, exactly what it stores, the precise memory formula, a worked example sizing a real model’s cache in gigabytes, why decode is bandwidth-bound rather than compute-bound, how the cache grows linearly with context and batch until it caps how many requests fit at once, how it is laid out in memory, and how MQA/GQA, paged attention, and quantization each pull a specific term of the formula down.
The redundancy the cache deletes
Autoregressive decoding generates one token at a time, and each step runs a full forward pass over the sequence so far. Attention at position t needs the keys and values of every earlier position: the new query q_t is dotted against all keys K = [k_1, …, k_t], softmaxed, and used to weight all values V = [v_1, …, v_t]. Those earlier keys and values are computed by projecting each token’s hidden state through W_K and W_V — and here is the catch: for a fixed past token, its key and value never change as generation continues.
So a cache-free implementation is absurdly wasteful. To emit token t it re-projects all t tokens through W_K and W_V; to emit token t+1 it re-projects all t+1 again, and so on. Generating a sequence of length N repeats the key/value projections roughly 1 + 2 + … + N ≈ N^2 / 2 times — quadratic work to produce a linear number of tokens. The KV cache exists to do each projection exactly once and remember the result.
What the cache actually stores
The cache stores, for every past token, its key and value vectors — and nothing else. Concretely, at each transformer layer and for each attention head, every token contributes one key vector of dimension d_head and one value vector of dimension d_head. The cache is therefore a set of tensors shaped, per layer, as [batch, n_kv_heads, seq_len, d_head] for keys and an identical one for values.
When a new token arrives during decode, the model computes only its query, key, and value — three projections of the single new hidden state. The new key and value are appended to the cached tensors (the seq_len axis grows by one), and the new query attends against the whole, now-extended cache. That is the entire mechanism: append one column of K and one column of V per step, then read the full history back. The cache turns the per-step cost from ‘re-project everything’ into ‘project one token, then read a growing buffer,’ which is exactly why decode shifts from being compute-heavy to being memory-heavy.
Why keys and values but not queries
A natural question: attention uses queries, keys, and values — why cache only two of the three? Because the query is transient. The query q_t is used exactly once, at step t, to attend against the accumulated keys and values, and is then never needed again. Past queries play no role in any future step — token t+5 attends with its own query q_{t+5}, not with q_t. There is nothing to remember.
Keys and values are the opposite: they are consumed by every future query. Token t’s key and value participate in the attention of steps t, t+1, t+2, and so on to the end of the sequence. Their reuse count grows without bound, which is precisely what makes caching them pay off. This asymmetry — queries are write-once-read-once, keys and values are write-once-read-many — is the reason the structure is a KV cache and not a QKV cache, and it is baked directly into the causal structure of decoder attention.
Deriving the memory formula
Now count bytes. Fix one transformer layer. For a batch of batch sequences, each of length seq_len, with n_kv_heads key/value heads each of dimension d_head, the key tensor holds batch × n_kv_heads × seq_len × d_head elements. The value tensor holds exactly the same. That factor of two — one copy for K, one for V — is the leading constant in every KV-cache estimate.
Multiply by n_layers because each layer keeps its own independent cache, and by bytes_per_elem to turn element counts into bytes. The full expression is:
KV_bytes = 2 × n_layers × n_kv_heads × d_head
× seq_len × batch × bytes_per_elemRead the terms as levers. The 2 is K-and-V and is fixed. n_layers, n_kv_heads, and d_head are architectural constants of a given model. seq_len and batch are set at runtime by your workload. bytes_per_elem is a precision choice. Every KV-cache optimization in existence works by shrinking one of these terms — which is why deriving the formula is worth the trouble.
Bytes per element and precision
bytes_per_elem is set by the numeric type the cache is stored in. Half precision — FP16 or BF16 — is the common default for inference and uses 2 bytes per element. FP32 would use 4, doubling the cache, and is rarely used for the KV cache in practice. Eight-bit formats (INT8, FP8) use 1 byte, halving it; four-bit KV quantization pushes toward 0.5 bytes.
It is worth being explicit that this is a genuinely free-floating knob: the KV cache does not have to share the precision of the model weights or the activations. A model can run its matmuls in BF16 while storing keys and values in INT8, because the cache is just a lookup buffer that gets dotted against queries — a small amount of quantization noise there is often tolerable. Because bytes_per_elem multiplies the entire formula, halving it halves the cache outright, with no change to any architectural term. That leverage is why KV-cache quantization is one of the highest-yield memory optimizations in serving, and it is treated as a sibling topic later.
A worked example: sizing Llama-2-7B
Take Llama-2-7B, which uses standard multi-head attention: n_layers = 32, n_kv_heads = 32, d_head = 128, stored in FP16 so bytes_per_elem = 2. First find the cost of a single token, at batch = 1:
per_token = 2 × 32 × 32 × 128 × 2 bytes
= 524,288 bytes
= 512 KiB / tokenHalf a megabyte per token, per sequence. Now fill the context window to seq_len = 4096:
KV_bytes = 512 KiB × 4096
= 2,147,483,648 bytes
= 2^31 bytes = 2 GiB ≈ 2.15 GBSo one full 4096-token conversation with a 7B model costs 2 GiB of cache — on top of the ~13 GiB the FP16 weights already occupy. (Throughout, 1 GiB = 2^30 bytes; the decimal GB = 10^9 value is quoted alongside so the units are never ambiguous.) That the answer lands on a clean power of two is a coincidence of these dimensions, but the magnitude is the point: the cache is not a rounding error next to the weights — it is the same order of magnitude.
Linear growth with context length
In the formula, seq_len appears to the first power, so the cache grows linearly with context. This is a crucial and sometimes surprising contrast with attention compute, which is O(seq_len^2). The KV cache is O(seq_len): doubling the context doubles the cache, it does not quadruple it.
Using the Llama-2-7B number of 512 KiB/token, the memory ladder is easy to read off. A 4K context is 2 GiB; 8K is 4 GiB; 32K is 16 GiB; 128K is a full 64 GiB — for a single sequence. This is why long-context serving is dominated by KV-cache memory rather than by attention FLOPs: the compute per decode step is modest, but the buffer you must hold and stream grows one token-slice at a time and never shrinks until the sequence ends. Long context is, from the memory system’s point of view, simply a very large and continuously growing KV cache. It is also why an idle-but-open long conversation is expensive: even between tokens, that 64 GiB has to stay resident, or be paged out and back, for the request to continue.
Linear growth with batch, and the concurrency cap
batch also enters to the first power, and it is the term that turns KV-cache math into a hard capacity limit. Each concurrent request carries its own independent cache; serving B requests at once means B copies. With Llama-2-7B at a 4K context, one request is 2 GiB, so a batch of 32 is 32 × 2 = 64 GiB of cache alone.
Put that on an 80 GiB accelerator: ~13 GiB goes to weights, leaving ~67 GiB for cache, which fits about 67 / 2 ≈ 33 concurrent 4K sequences and no more. Push the context to 32K and each request needs 16 GiB — now barely four concurrent requests fit. This is the real ceiling on serving throughput: not compute, but how many KV caches fit in memory simultaneously. Maximum batch size — and therefore token-per-second throughput and cost-per-token — is dictated by (memory − weights) / KV_bytes_per_request. Every GiB you save per request is directly one more request you can serve at once, which is the entire commercial motivation behind the mitigations.
Why decode is memory-bandwidth-bound
The cache reshapes not just how much memory decode uses but what its bottleneck is. Consider a single decode step at batch 1. The model does a handful of matrix-vector products — one new token’s hidden state against the weight matrices — plus attention against the cache. The arithmetic is tiny: a vector, not a matrix, flows through each weight. But to do it, the hardware must read every weight and read the entire KV cache from memory.
That ratio — bytes moved versus FLOPs performed, the arithmetic intensity — is what decides the bottleneck. Prefill processes many tokens at once, so each loaded weight is reused across the whole prompt: high intensity, compute-bound. Decode processes one token, so each loaded weight is used for a single vector-multiply and thrown away: low intensity, memory-bandwidth-bound. The step time is set by how fast you can stream the weights and the KV cache past the compute units, not by the compute itself. This is why decode latency tracks memory bandwidth, why a bigger KV cache directly slows generation (more bytes to stream per token), and why batching helps — it amortizes the weight reads across more sequences.
The cache versus the weights: the memory budget
It helps to hold the two big consumers of inference memory side by side. Model weights are a fixed cost: Llama-2-7B in FP16 is about 13 GiB whether you serve one request or fifty, and whether the context is 100 tokens or 100K. The KV cache is a variable cost that scales with seq_len × batch and can dwarf the weights.
At a 4K context, 33 concurrent requests already push the cache to ~66 GiB — five times the weight memory. This inversion is the defining feature of high-throughput serving: past a modest batch size, you are not running a 13 GiB model, you are running a 13 GiB model wrapped in a 60 GiB, constantly-churning cache. Capacity planning for inference is therefore mostly KV-cache planning. You budget weights once, then spend everything left on caches, and the shape of your workload — long contexts versus many short ones — decides whether you are memory-starved on seq_len or on batch. Getting this arithmetic right is the difference between an accelerator that serves four users and one that serves forty.
Physical layout and fragmentation
The formula counts bytes; layout decides how efficiently those bytes are actually used, and it is the third pillar of the topic. The naive implementation gives each sequence one large, contiguous buffer sized to the maximum supported context — because appending a token must not require moving the whole cache, so the space is reserved up front. A request allowed up to 4K tokens gets a 4K-token slot immediately, even while it is only 200 tokens long.
The result is severe internal fragmentation. Every active sequence pads its cache out to the maximum, so a server full of short conversations wastes most of its reserved KV memory on empty, pre-allocated slots. Worse, contiguous per-sequence allocation causes external fragmentation too: freed slots of the wrong size cannot be reused, and the effective batch size falls well below what the raw formula predicts. In measured systems this waste routinely ran to 60–80% of KV memory. Recognizing that the enemy is not the byte count but the reservation-to-the-max, contiguous layout is exactly what motivates paged attention, which stores the cache in small fixed-size blocks like an operating system pages virtual memory.
A GQA contrast: shrinking n_kv_heads
The Llama-2-7B example used multi-head attention, where every query head has its own key/value head, so n_kv_heads = n_heads = 32. Grouped-query attention (GQA) breaks that coupling: many query heads share one key/value head, so n_kv_heads drops sharply while n_heads stays the same. Because n_kv_heads is a direct multiplicative term in the cache formula, this is a clean, large reduction.
Llama-3-8B is the concrete contrast: n_layers = 32, d_head = 128, but n_kv_heads = 8 instead of 32. Its per-token cost is 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB/token — exactly one quarter of Llama-2-7B’s 512 KiB. An 8K context is 1 GiB instead of 4 GiB. The same 80 GiB accelerator that held ~33 short-context 7B requests now holds four times as many, at essentially no quality cost. This is why nearly every recent model ships with GQA: it is the cheapest term in the formula to attack, and it attacks it by a fixed integer ratio.
Mitigations (siblings)
Every KV-cache optimization is best understood as pulling one specific term of 2 × n_layers × n_kv_heads × d_head × seq_len × batch × bytes_per_elem downward. Four families matter, each a sibling topic in this series:
MQA and GQA shrink n_kv_heads. Multi-query attention takes it all the way to 1 (all query heads share a single K/V head); grouped-query attention picks an intermediate group count — the Llama-3-8B case above — trading a little quality for a fixed-ratio cut. KV-cache quantization shrinks bytes_per_elem, storing keys and values in INT8 or FP8 (or lower) independent of the compute precision, halving or quartering the whole cache. Paged attention does not shrink any term — it attacks the layout, replacing the contiguous max-length reservation with small fixed-size blocks so the fragmentation described earlier nearly vanishes and the effective batch approaches what the formula promises. A fourth lever, eviction and sliding windows, caps the effective seq_len by dropping or compressing old tokens. Each is a full topic; the point here is that the formula tells you exactly which term each one targets.
Practical implications and pitfalls
A few consequences fall straight out of the math. First, on a memory-constrained CPU or small-GPU deployment — the CPU-SLM regime — the KV cache, not the weights, is usually what caps your usable context and concurrency, so a smaller-context or GQA model can be the difference between fitting and not. Second, KV-cache precision is a separate knob from weight precision; quantizing the cache to INT8 is often a larger memory win than shaving the weights, and independently tunable. Third, throughput is a memory-capacity question before it is a compute question — if generation feels slow, look at bandwidth and cache size, not FLOPs.
The common pitfalls: forgetting the leading factor of two and underestimating the cache by half; using n_heads instead of n_kv_heads for a GQA/MQA model and overestimating by the group ratio; conflating GB and GiB and being off by 7%; and — the operational one — budgeting only for the average context length when a handful of max-length requests can exhaust KV memory and stall the whole batch. Size for the tail, in the right units, with the right head count.