Why architecture matters here

Halving KV cache means doubling context or throughput at similar cost — a real serving win. But quantization is easy to get wrong: outliers in K or V can wreck int8 fidelity, RoPE angles amplify quantization error, and a naive kernel dequant negates the memory savings via increased compute.

The architecture matters because each subsystem depends on the others. The quantizer needs calibration data; the kernels need dequant fused with attention; the eval harness needs long-context tasks to catch regressions.

With the full pipeline in mind, KV cache quantization becomes a reliable knob rather than a risky science project.

Advertisement

The architecture: every piece explained

The top strip is the quantization pipeline. Attention layer produces K and V tensors during prefill and decode. Calibration set is a set of representative prompts used to profile activation ranges. Quantizer computes scales and zero points from calibration statistics. KV cache store holds the compressed int8 or int4 blocks — often paged for memory efficiency.

The middle row is the techniques. Per-channel scale uses one scale per head dimension — captures channel outliers. Per-token group uses grouped quantization along the sequence axis — isolates token-level outliers. RoPE handling is critical: quantize AFTER RoPE rotation because RoPE mixes signals across dimensions and quantizing before amplifies error. Dequant at attn means the kernel dequantizes on-the-fly during softmax(QK) computation — no separate materialization step.

The lower rows are integration. Kernel support requires fused int4/int8 attention kernels (e.g., Marlin, custom TRT-LLM). Eval harness validates on perplexity, long-context recall, and downstream task suites. Serving integration documents the memory-vs-quality trade per model and workload so operators know when to enable.

KV cache quantization — fp16 → int8/int4 with per-token / per-channel scalingshrink the biggest memory pool in LLM servingAttention layerproduces K,VCalibration setrepresentative promptsQuantizerscale + zero-pointKV cache storeint8 / int4 blocksPer-channel scaleper head dimPer-token groupoutlier isolationRoPE handlingrotate before quantDequant at attnon-the-fly to fp16Eval harnessppl + long-context + task suiteKernel supportfused int4 attentionServing integration — memory savings vs quality trade documented per model + workloadproduceprofilecompressstorefuseverifydequantintegrateintegrate
KV cache quantization pipeline and integration surface.
Advertisement

End-to-end flow

End-to-end: a serving team wants to enable int4 KV cache quantization on a 70B model at 32k context. They gather a calibration set of 1000 diverse prompts, run through the model, and compute per-channel scales for K and V post-RoPE. The quantizer stores scales alongside the KV cache. Inference kernels use fused int4 attention: K and V are dequantized during the softmax kernel. The eval harness runs perplexity (0.01 point delta), needle-in-haystack at 32k (0.02 recall drop), and a task suite (within noise). Memory dropped 4x; concurrency doubled at the same latency. The team ships with a flag to fallback to fp16 KV cache if regressions appear.