Why it matters
KV cache is the single biggest optimization in LLM inference. Also the biggest memory consumer. Understanding it explains inference cost and shapes serving architecture.
The architecture
During generation of token t: compute Q, K, V for new token. Concatenate new K, V to cached K, V from previous tokens. Attend using new Q over full cached K, V.
Cache grows with each token; contains K and V for every generated position across every layer and head.
How it works end to end
Memory cost: 2 (K + V) × sequence length × hidden dim × num layers × dtype size. For 30B model at 8K context: hundreds of MB per sequence.
Batch serving: cache per sequence; total memory scales with batch size and context length.
PagedAttention: manages cache as pages like OS memory management. Enables efficient batching with variable-length sequences.
GQA/MQA: reduces cache size by sharing K, V across query heads.