KV Cache Quantization — Bytes per Token and int8/int4 Trade-offs

What the KV cache is, and why it dominates decode

Attention lets every token look back at every earlier token. During autoregressive decoding you generate one token at a time, and recomputing the keys and values for the entire history at each step would be quadratic waste. So you cache them: for each past position, each layer, and each key/value head, you keep the projected K and V vectors. Every new token computes its query, dots it against all cached keys, softmaxes, and takes a weighted sum of all cached values. The cache turns a quadratic recompute into a linear read.

The catch is that the cache is state, and it grows one slab per token, forever, for the life of the sequence. Weights are fixed no matter how long you decode; the KV cache scales with context length × batch size. That is why, in long-context serving, the cache — not the weights — is usually the binding memory constraint, and why shrinking it is one of the highest-leverage optimizations in inference. Quantization is the most direct lever: change the bytes per stored number and the whole slab shrinks in proportion.

The bytes-per-token formula

Size the cache exactly. For one token, one layer stores a key vector and a value vector, one of each per key/value head, each of width d_head. Across the whole model that is:

bytes_per_token = 2 × n_layers × n_kv_heads × d_head × bytes_per_elem

        2         → one K and one V
  n_layers        → cache is per-layer
  n_kv_heads      → KV heads, NOT query heads (this is where GQA/MQA bite)
  d_head          → width of each head
  bytes_per_elem  → fp16=2, int8=1, int4=0.5

Total cache is then bytes_per_token × seq_len × batch. Two features of this formula matter for quantization. First, it is perfectly linear in bytes_per_elem — halving the bit-width halves the cache, no asterisks on the leading term. Second, the n_kv_heads factor (not the query-head count) is exactly what grouped-query attention already shrinks; quantization multiplies on top of that saving rather than competing with it. Everything downstream is arithmetic on this one line.

Worked example: fp16 vs int8 vs int4

Take a Llama-3-70B-shaped model: n_layers = 80, n_kv_heads = 8 (it uses GQA), d_head = 128. Per token:

elems_per_token = 2 × 80 × 8 × 128 = 163,840 numbers

  fp16 (2 B):   163,840 × 2   = 327,680 B  ≈ 320 KB / token
  int8 (1 B):   163,840 × 1   = 163,840 B  ≈ 160 KB / token
  int4 (0.5 B): 163,840 × 0.5 =  81,920 B  ≈  80 KB / token

Now stretch to a 32K-token context at batch = 1:

  fp16:  320 KB × 32,768 ≈ 10.0 GB
  int8:  160 KB × 32,768 ≈  5.0 GB
  int4:   80 KB × 32,768 ≈  2.5 GB

That 7.5 GB swing between fp16 and int4 is the difference between one long conversation crowding a 24 GB GPU and four of them fitting comfortably. In batched serving the effect compounds: since the cache scales with batch, int4 lets you pack 4× as many concurrent long-context sessions into the same memory. (These are raw payload figures; quantization scales and zero-points add a small overhead we account for later.)

Decode is memory-bound, so smaller is also faster

Shrinking the cache saves memory, but there is a second prize. Token-by-token decode is memory-bandwidth-bound, not compute-bound: each step does very little arithmetic (one new query against the history) and instead spends its time reading the weights and the entire KV cache out of memory. The bottleneck is bytes moved per token, not FLOPs.

So halving the cache's byte-width roughly halves the bytes the attention kernel must stream for the cache portion of each step, which can translate into faster decoding — especially at long context where the cache read dominates. The caveat is that the numbers must be dequantized back to a compute type before the dot products, so the kernel pays a small unpacking cost; a well-written fused kernel does the dequant on-chip and still comes out ahead because memory traffic, not the multiply, was the limiter. The upshot: KV quantization is one of the rare optimizations that improves the memory ceiling and the latency floor at the same time, which is exactly why it earns its place on memory-starved and CPU-SLM deployments.

Integer quantization in one screen

Quantization maps a range of real values onto a small set of integers. Fix a scale s (how much real value each integer step is worth) and, optionally, a zero-point z (which integer represents real zero). The round trip is:

quantize:    q = round(x / s) + z      then clamp to the integer range
dequantize:  x’ = (q - z) × s

int8  range: [-128, 127]  (signed)  or  [0, 255]  (unsigned)
int4  range: [-8, 7]      (signed)  or  [0, 15]   (unsigned)

The error on any value is at most half a step, ±s/2, so the whole game is choosing s well: too large and you waste precision on a range no value occupies; too small and large values clip to the range limit, which is far more damaging than rounding. Because int4 has only 16 levels versus int8’s 256, its step s is roughly 16× coarser for the same range — the reason int4 needs much more careful scale placement to stay accurate. Everything that follows — symmetry, granularity, outlier handling — is about picking s (and z) so the clamp never fires on values that matter and the steps land where the numbers actually live.

Symmetric vs asymmetric

The zero-point is the knob that distinguishes the two schemes. Symmetric quantization sets z = 0 and picks the scale from the largest magnitude in the group: s = max(|x|) / qmax. It is cheap — dequant is a single multiply, and the integer zero maps to real zero, which keeps sparse or zero-centered tensors exact. Its weakness is skew: if a tensor’s values run, say, from -0.1 to +2.0, a symmetric range of ±2.0 wastes half its codes on negatives that never appear.

Asymmetric quantization fits the range to the actual min and max, s = (max - min) / (qmax - qmin), and uses z to shift so the smallest real value lands on the smallest integer. It spends every code on the range that is occupied, which matters a lot for skewed distributions — and key/value activations are frequently skewed. The cost is a slightly heavier dequant (subtract z, then multiply) and one extra stored number per group. For KV caches, where values are activations with no reason to be symmetric about zero, asymmetric quantization usually buys real accuracy, which is why aggressive schemes (int4 and below) lean on it.

Granularity: per-tensor, per-token, per-channel

A single scale for an entire tensor is cheapest but crudest: one outlier anywhere inflates the range for everyone, coarsening every step. Finer granularity gives each slice its own s (and z), so a slice with a modest range gets fine steps regardless of what other slices do. For a KV cache, laid out as [tokens, heads, d_head], two axes matter:

The practical sweet spot is group-wise: split each vector into contiguous groups of, say, 64 or 128 elements and quantize each group independently. It localizes any outlier’s damage to its own small group while keeping the scale-storage overhead low. But which axis you group along turns out to matter enormously for the cache, and that is the outlier story.

The outlier problem: keys have outlier channels

Quantization’s enemy is the outlier — a handful of values far larger than the rest that stretch the range and coarsen every step for the whole group. In transformer activations these are not random; they are structured. The empirically important finding (from work like KIVI) is that the key cache carries outliers concentrated in specific channels — certain positions along the d_head axis are persistently large across essentially all tokens — while the value cache shows no such fixed-channel structure.

This has a sharp consequence for granularity. If you quantize keys per-token (across the channel axis), every token’s scale is dragged up by those same outlier channels, wrecking the precision of all the well-behaved channels. If instead you quantize keys per-channel, an outlier channel gets its own large scale and simply stops contaminating its neighbors. The outliers are isolated by construction. Getting this axis right is the single biggest determinant of whether low-bit KV quantization preserves quality — quantize the keys along the wrong axis and int4 falls apart; quantize along the channel axis and it holds up remarkably well.

Why keys and values want different axes

The natural design that falls out is per-channel keys, per-token values. Keys go per-channel to quarantine those fixed outlier channels. Values go per-token because they have no channel structure to exploit, and because the value side has a second, mechanical reason to prefer the token axis.

Recall how each is consumed. The attention score is a dot product of the query with a key across the d_head channel axis, so grouping keys along channels aligns cleanly with how they are reduced. The output, by contrast, is a weighted sum of value vectors across the token axis, so a per-token scale on values factors neatly out of that sum. There is also a streaming reason: new tokens arrive one at a time, and a per-token value scale can be computed on the spot when the token is appended, with nothing to recompute later. Per-channel key scales need a little more care — a fixed channel statistic, or a running estimate, or a small full-precision residual buffer of the most recent tokens — but the payoff, clean int4 or even 2-bit keys, is worth the bookkeeping.

The accuracy you actually pay

How much quality does this cost? The honest answer is it depends on the bit-width and the care taken, but the shape is consistent. int8 KV cache is close to free: with per-token or per-channel scales it is routinely reported as near-lossless on perplexity and downstream tasks, to the point that many serving stacks offer it as a low-risk default. The 2× saving comes with a difference you generally cannot measure on benchmarks.

int4 is where care starts to matter. With naive per-tensor or wrong-axis quantization it degrades visibly, especially on long-context and reasoning tasks that are sensitive to small attention perturbations. With the right recipe — per-channel keys, per-token values, group-wise scales, often a handful of recent tokens kept in full precision — int4 KV cache lands within a small margin of fp16 on most workloads. Below int4 (2-bit and 3-bit) is an active research frontier: achievable, but only with the full bag of tricks. A useful rule: budget int8 as safe, int4 as safe-with-a-good-kernel, and anything lower as measure-before-you-trust.

Symmetric or asymmetric for KV, concretely

Putting symmetry together with the outlier picture gives practical defaults. Value activations are typically not centered on zero and have no reason to be symmetric, so asymmetric quantization — fitting the scale and zero-point to the real min and max of each token’s value vector — spends codes where the data is and tends to win, particularly at int4 where every code is precious.

Keys are more nuanced. Once you have committed to per-channel scales that isolate the outlier channels, each channel’s distribution is far tamer, and symmetric quantization on those isolated channels is often good enough and cheaper — the single multiply keeps the fused attention kernel simpler and faster. So a common shape is asymmetric per-token values and symmetric per-channel keys, though plenty of systems use asymmetric on both. The meta-point is that symmetry is a second-order knob: get the axis right first (that is what tames the outliers), then choose symmetry to trade a little accuracy against kernel simplicity. Reaching for asymmetry to fix an outlier problem that is really an axis problem just adds cost without curing the disease.

Scales and zero-points: the honest bit-width

Quantization is never exactly 4 bits per number, because each group also stores its scale (and, if asymmetric, its zero-point), usually in fp16. Those extras are amortized across the group, so the effective bit-width depends on group size:

eff_bits = base_bits + (metadata_bits / group_size)

int4, symmetric,  group 128:  4 + 16/128      = 4.125 bits/elem
int4, asymmetric, group 128:  4 + (16+16)/128 = 4.25  bits/elem
int4, asymmetric, group 32:   4 + 32/32       = 5.0   bits/elem

The tension is immediate: smaller groups isolate outliers better and improve accuracy, but their metadata overhead grows until, at group 32, an asymmetric int4 cache is really costing 5 bits — you have handed back a quarter of the saving. This is why group 128 is such a common default: it captures most of the accuracy of fine grouping while keeping overhead near a rounding error. When you quote ‘int4 KV cache = 4× smaller,’ the truthful figure is nearer 16/4.25 ≈ 3.76× — still enormous, but worth stating honestly when you are budgeting memory to the gigabyte.

How it composes with GQA and MQA

KV quantization and grouped-query attention attack the same term in the bytes-per-token formula — and, crucially, they multiply rather than overlap. GQA and MQA shrink the n_kv_heads factor: multi-head attention gives every query head its own KV head, GQA shares one KV head across a group of query heads, and MQA takes that to the limit with a single KV head for all of them. Our 70B example already reflects GQA — 8 KV heads feeding 64 query heads, an 8× cache reduction versus full multi-head before a single bit is quantized.

Quantization then scales bytes_per_elem, an orthogonal factor. Stack them and the savings compound: relative to fp16 multi-head, GQA’s 8× times int4’s 4× is a 32× smaller cache. This is why modern long-context serving leans on both at once — architectural sharing to cut how many K/V vectors exist, then quantization to cut how many bytes each one costs. Because they touch different factors, adopting one never diminishes the return on the other; the design question is simply how far to push each before quality gives way.

How it composes with paged attention

Paged attention (the idea behind vLLM) solves a different problem: fragmentation. Instead of reserving one big contiguous cache buffer per sequence — which wastes memory whenever a sequence ends short of its maximum — it chops the cache into fixed-size blocks and allocates them on demand, like virtual-memory pages, so blocks from many sequences pack tightly and can even be shared across requests with a common prefix. It raises the utilization of whatever cache memory you have.

Quantization is fully complementary: it shrinks the content of each block, while paging manages the placement of blocks. A quantized paged cache simply stores int8 or int4 payloads in each page rather than fp16, so more tokens fit per page and more pages fit in memory — the two savings multiply. The one detail to respect is that quantization metadata (the per-group scales and zero-points) has to be paged alongside its data so a block stays self-describing when it moves. Get that bookkeeping right and you inherit both wins at once: paging removes the waste between sequences, quantization removes the waste within each stored number.

CPU and small-model implications

On a CPU or a small-model edge deployment the case for KV quantization is, if anything, stronger than on a big GPU. Memory is tighter and, more importantly, memory bandwidth is the dominant constraint on CPU decode: there are no thousands of parallel arithmetic units to hide behind, so the time per token is largely the time to stream weights and cache through a comparatively narrow memory bus. Halving the cache’s byte-width directly attacks the thing the CPU is waiting on.

There is a pleasant synergy with weight quantization too. A small model whose weights are already int4 or int8 pairs naturally with an int8 or int4 KV cache: the compute type and the storage types line up, dequant paths are already present in the kernels, and the whole footprint — weights plus cache — can be made to fit a modest RAM budget that fp16 would blow through. For a long-context assistant running locally, an int4 cache can be the difference between holding a full document in context and truncating it. Start at int8 (near-free), reach for int4 when the context or the concurrency demands it, and measure quality on your own prompts before trusting anything lower.

Pitfalls and a practical checklist

KV quantization is high-leverage but has sharp edges. The most common mistake is quantizing keys along the token axis, which lets the fixed outlier channels ruin every token’s scale — the fix is per-channel keys. A close second is chasing accuracy with tiny groups until the scale/zero-point overhead has eaten much of the saving; check the effective bit-width, not the nominal one. Watch for clipping: an under-sized range that clamps large values hurts far more than rounding, so calibrate ranges on representative data, and consider keeping the few most recent tokens in full precision, since those attend most sharply and are least forgiving of error.

A working checklist: (1) start at int8, treat it as a near-free default; (2) for int4, use per-channel keys and per-token values; (3) prefer group-wise scales around 128 elements; (4) use asymmetric quantization on values, symmetric-or-asymmetric on the already-tamed key channels; (5) account for metadata in your memory budget; (6) verify the dequant is fused into the attention kernel so you keep the bandwidth win; and (7) always measure quality on your own long-context workload before shipping anything below int4. Do these and the cache stops being the wall.