KV Cache Quantization

Why it matters

KV quantization halves decode memory. Big win for long-context serving.

The architecture

Store K, V in INT8/INT4.

Dequantize on read.

Slight quality loss.

How it works end to end

Per-token scale/zero.

Dequantize in attention.

Enable in vLLM, TRT-LLM.