Why it matters

KV cache is memory bottleneck. Understanding optimizations enables serving longer context / more concurrent.

Advertisement

The architecture

Architecture: GQA/MQA reduce K/V heads.

Quantization: KV in lower precision.

KV cache optimizationsGQA / MQAarch: fewer KV headsQuantizationINT8 / INT4 KVEvictiondrop old blocksGQA is architectural (fixed by model); quantization + eviction are inference-time
KV optimization types.
Advertisement

How it works end to end

GQA (grouped query attention): fewer K/V heads shared by more Q heads. Modern models (Llama 3, Mixtral) use it.

KV quantization: to INT8 or INT4. Some quality loss, big memory savings.

Eviction: for long contexts, drop old/less-important KV blocks.

Compression: research-stage.