Why architecture matters here
The economics of LLM serving are memory-bound at the frontier of context length. Autoregressive generation is fundamentally a memory-movement problem: each new token reads the entire KV cache to compute attention, so serving throughput is gated by how much cache you can hold and how fast you can move it. The KV cache's size grows without bound as conversations lengthen — a 100K-token context can require tens of gigabytes of cache for a single sequence on a large model — and because it is per-sequence, batching many concurrent users multiplies it. This is why a deployment often hits an out-of-memory wall not on weights but on cache, and why the cache is the highest-leverage place to save memory.
Quantizing the cache is high-leverage precisely because the savings convert directly into the two things serving cares about: context length and batch size. Halving the per-token cache with int8 lets you either double the maximum context a single request can reach, or double the number of requests you batch at a given context — either way roughly doubling the useful work per GPU. int4 pushes that to 4x. Because the cache is often the dominant memory consumer at long context, these multipliers on the cache translate almost one-to-one into multipliers on serving capacity, which is a far larger lever than shaving a few percent off compute.
What makes KV-cache quantization a genuine architecture problem — not just 'use fewer bits' — is that the cache is far more quality-sensitive than the weights, and the sensitivity is uneven. The keys and values are activations, not parameters, and they contain outliers: a few channels with large magnitudes that carry disproportionate information, and crushing them into a coarse quantization grid degrades output quality quickly. Empirically keys are more sensitive than values — keys feed the softmax where small errors get amplified through the exponential, while values are averaged and more forgiving. So a naive uniform int4 on everything can wreck a model that tolerates int8-keys/int4-values or per-channel key quantization just fine. The architecture is in choosing what to quantize how, not merely how far.
The final architectural constraint is that the compression must not cost more than it saves. Every read of the cache now needs a dequantization step before the attention dot-product, and if that step is a separate memory-bound pass it can erase the throughput win — you saved memory but added latency. So real implementations fuse dequantization into the attention kernel, dequantizing on the fly as they stream the cache through the matmul, so the quantized cache is read in its compact form (less memory bandwidth, the actual bottleneck) and expanded only in registers. Getting this kernel right is what turns KV-cache quantization from a memory trick into a throughput win, because moving fewer bytes through the memory system is itself faster, not just smaller.
The architecture: every piece explained
Top row: where the cache comes from and how it is compressed. The model's attention layers produce, for every token, a key and a value vector per head; normally these are stored in the KV cache in fp16, and that cache grows with sequence length times layers times heads. KV-cache quantization inserts a quantize-on-write step: as each token's K and V are produced, they are compressed from fp16 to int8, fp8, or int4 before being written to the cache. To make that reversible, the serving engine stores scales (and zero-points for asymmetric schemes) alongside the quantized data — computed per-token (one scale per token vector) or per-channel (one scale per feature dimension), the choice that most affects quality.
Middle row: reading the cache back for attention. The compressed cache is the win — a compact cache that holds two to four times as many tokens in the same bytes. When a new token attends, the stored quantized keys and values are dequantized on read using their scales, reconstructing an approximation of the original fp16 vectors, which feed the attention score computation (Q·Kᵀ, softmax, then the weighted sum over V). The whole point of the pipeline is that everything downstream of the cache is unchanged — the model still computes attention in higher precision — only the stored form is compressed. A quality check (perplexity or task-eval delta versus the fp16 baseline) is how you confirm the compression didn't cross the line into unacceptable degradation.
Bottom-left: the accuracy-preserving tricks. Because activations have outliers — a few high-magnitude channels that dominate — good schemes handle them specially: keep the most sensitive dimensions (or the keys entirely) in higher precision, quantize per-channel so an outlier channel gets its own scale rather than blowing up a shared one, and sometimes keep the most recent tokens in full precision since they matter most. Bottom-right: the kernel is where performance is won or lost — a fused dequantize-plus-matmul reads the cache in compact form (saving the memory bandwidth that is the real bottleneck) and dequantizes in registers on the fly, so quantization speeds attention up rather than adding a separate slow pass.
Bottom strip: the operational surface. KV-cache quantization is a dial you must tune per model, so the numbers you watch are the bits-versus-quality trade (how far you can drop precision before eval degrades past your threshold), the throughput/capacity gain (the actual context-length or batch-size multiplier realized, which the kernel determines), and per-model calibration (the scales and outlier choices that work for one model architecture rarely transfer unchanged to another, so each model needs its own validation).
End-to-end flow
Trace a long-document assistant serving a 32K-token context on a single GPU. In fp16 the KV cache for one such request consumes, say, 20 GB — leaving little room for the weights and forcing a batch size of one, so the GPU serves a single user at a time and its compute sits underutilized waiting on memory. The team enables int8 KV-cache quantization. Now each token's key and value are quantized to int8 on write with a per-token scale stored alongside, halving the cache to 10 GB. Immediately the GPU can either serve a 64K context for one user or batch two 32K users — doubling useful capacity with the weights untouched.
Follow one token through the compressed path. The model generates token 5,000; its attention layer produces a key and value vector per head in fp16. The quantize-on-write step computes a scale for each vector (per-token quantization: the max absolute value sets the int8 range), divides and rounds the vector into int8, and writes the int8 data plus the fp16 scale into the cache. The original fp16 vectors are discarded. Memory used for this token's cache entry: half of fp16.
Now token 5,001 must attend to all 5,000 prior tokens. Instead of reading 5,000 fp16 key vectors, the fused attention kernel streams the 5,000 int8 key vectors — half the bytes across the memory bus, which is the actual bottleneck — and dequantizes each in registers using its stored scale just before the Q·Kᵀ dot-product. The softmax and the weighted sum over the (also int8, dequantized-on-read) values proceed exactly as in fp16. The output token is computed in full precision; only the stored keys and values were ever compressed. Because fewer bytes moved, this attention step is actually a touch faster than fp16, on top of using half the memory.
Now the quality edge. The team first tried int4 on both keys and values to get a 4x saving, and the assistant started producing subtly worse summaries — the perplexity delta versus the fp16 baseline crossed their threshold, and a human eval confirmed degraded coherence on long inputs. Investigating, they found the culprit was the keys: a handful of key channels had large outlier magnitudes, and int4's coarse grid crushed them, corrupting the softmax where those channels dominated the attention scores. The fix was asymmetric: keep keys at int8 with per-channel scales (so each outlier channel gets its own range) and quantize only the more-forgiving values to int4. That combination held perplexity within tolerance while still cutting the cache substantially — a concrete instance of the general rule that keys are more sensitive than values and per-channel scaling is what tames outliers.
The last operational lesson is that none of these settings transfer for free. When the team deployed a second model — a different architecture with grouped-query attention and a different training recipe — the int8-keys/int4-values configuration that was safe on the first model degraded the second, because its activation distributions and outlier structure were different. They had to re-calibrate: sweep bit widths and per-channel-versus-per-token scaling, measure the perplexity and task-eval delta for each, and pick the most aggressive setting that stayed within the quality budget for that model. This per-model calibration is the recurring cost of KV-cache quantization: the memory win is large and reliable, but the exact bits-and-scaling recipe is a property of each model that must be measured, not assumed — which is why the quality-check step is a permanent part of the pipeline, not a one-time gate.