Why architecture matters here
KV quantization matters because KV cache size dominates memory for long-context serving. Weight memory is fixed; KV grows linearly with context and quadratically with concurrent request count. Cutting KV by 4x means 4x more concurrent long-context requests per GPU.
Cost impact is large. Long-context serving is often KV-memory-bound; more requests per GPU = lower cost per token.
Reliability of quality is where care is needed. KV quantization can hurt long-context reasoning tasks disproportionately. Task-specific eval matters.
The architecture: every step explained
Walk the diagram top to bottom.
FP16 KV cache. Standard 2-byte format. For 70B model at 128k context: ~40GB per request.
Choose format. INT8 for 2x savings with minimal quality loss; INT4 for 4x with more quality trade; FP8 for Hopper GPUs with hardware support.
Per-head scale. Granularity of quantization scales. Per-head is standard; per-token is finer; per-tensor is coarsest.
Write path. When new tokens' KV are produced, quantize before writing to cache. Store scale per group.
Read path. During attention, dequantize on-the-fly. Ideally fused into attention kernel to avoid materializing FP16.
Attention Kernel. Dequantize + gemm fused for speed. vLLM and TRT-LLM ship optimized kernels.
Accuracy check. Especially at long context, quantization can degrade reasoning. Eval task-specific.
Memory savings. INT8: 2x. INT4: 4x. Translates directly to concurrency capacity.
Hopper FP8 special case. H100+ have native FP8 tensor cores. FP8 KV runs at speed with near-FP16 quality.
Runtime support. vLLM (--kv-cache-dtype fp8), TensorRT-LLM (KV int8/int4/fp8), TGI. Enable per model config.
End-to-end KV-quant inference flow
Trace an inference. Serving 70B model with vLLM. Configure --kv-cache-dtype fp8.
Prompt of 8k tokens arrives. Prefill runs; produces KV cache. FP16 KV would be ~2.5GB for this request; FP8 KV is ~1.25GB.
Decode starts. Attention kernel reads KV, dequantizes FP8 to FP16 in registers, computes softmax + gemm. Result appended as new KV in FP8.
Concurrent requests pack more into the same GPU HBM. Instead of 8 concurrent 8k-context requests, you fit 16.
Quality eval: task-specific benchmark shows 0.3 pt drop on general QA (acceptable). Long-context needle-in-haystack shows minor accuracy loss at 128k (evaluate for your workload).
Switch to INT4 KV for extreme memory pressure. 4x savings; quality drop larger but for some tasks acceptable.
Metrics: tokens-per-second per GPU up 60%. Cost per token down proportionally.