KV Cache Deep Dive

Why it matters

KV cache is the memory workhorse of decode. Understanding size shapes serving cost.

The architecture

K, V stored per layer per head.

Decode reads all past K/V.

Grows linearly with tokens.

How it works end to end

Sized per layer × heads × dim.

Grows with tokens.

Read every step of decode.