Why it matters

KV cache dominates memory for long context. Quantization shapes deployability.

Advertisement

The architecture

Per-head + per-token scaling.

Quantize on write to cache.

Dequantize on read.

KV quantization flowFull precision K, VcomputeQuantizeint8 / int4Cachesmaller memoryQuality loss minimal at int8; int4 needs finer scaling schemes
KV quant.
Advertisement

How it works end to end

Per-head scaling: preserves head-specific stats.

Symmetric vs asymmetric.

Group quant (e.g., group of 128).