LLM Quantization Architecture in Depth

Why fewer bits make decoding faster

Start with why it works. Generating a token during decode means multiplying a small activation vector, one row per sequence in the batch, against every weight matrix in the model. At small batch sizes the GPU spends most of that time waiting for weights to stream out of HBM, not doing arithmetic. Decode is memory-bandwidth bound.

A back-of-envelope bound makes this concrete. An H100 SXM has roughly 3.35 TB/s of HBM bandwidth. Reading 141 GB of BF16 weights once takes about 42 ms, so a single sequence on hardware like that cannot exceed roughly 24 tokens per second no matter how fast the tensor cores are (and the model would actually need to be split across two GPUs). Cut the weights to about 40 GB and the same bound becomes about 12 ms per step. Real throughput is lower, but the direction holds: fewer bytes per weight, less time per decode step.

Prefill is different. Processing a 4,000-token prompt multiplies large matrices with high arithmetic intensity, so it is compute bound. Weight-only formats such as W4A16 still have to convert weights back to 16-bit before the tensor cores can use them, so they speed up prefill very little and can even slow it down. To make prefill faster you must lower the precision of the arithmetic itself, which means quantizing activations too (W8A8 in INT8 or FP8, or FP4 on hardware that supports it). This single distinction, bandwidth-bound decode versus compute-bound prefill, explains most format choices.

Why fewer bits make decoding faster

Start with why it works. Generating a token during decode means multiplying a small activation vector, one row per sequence in the batch, against every weight matrix in the model. At small batch sizes the GPU spends most of that time waiting for weights to stream out of HBM, not doing arithmetic. Decode is memory-bandwidth bound.

A back-of-envelope bound makes this concrete. An H100 SXM has roughly 3.35 TB/s of HBM bandwidth. Reading 141 GB of BF16 weights once takes about 42 ms, so a single sequence on hardware like that cannot exceed roughly 24 tokens per second no matter how fast the tensor cores are (and the model would actually need to be split across two GPUs). Cut the weights to about 40 GB and the same bound becomes about 12 ms per step. Real throughput is lower, but the direction holds: fewer bytes per weight, less time per decode step.

Prefill is different. Processing a 4,000-token prompt multiplies large matrices with high arithmetic intensity, so it is compute bound. Weight-only formats such as W4A16 still have to convert weights back to 16-bit before the tensor cores can use them, so they speed up prefill very little and can even slow it down. To make prefill faster you must lower the precision of the arithmetic itself, which means quantizing activations too (W8A8 in INT8 or FP8, or FP4 on hardware that supports it). This single distinction, bandwidth-bound decode versus compute-bound prefill, explains most format choices.

Number formats, scales and granularity

Every integer scheme maps a real value to a small integer code with a scale and, optionally, a zero point: q = clamp(round(x / s) + z, qmin, qmax) and back with x_hat = s * (q - z). Symmetric schemes fix z = 0; asymmetric schemes spend extra storage on z to cover skewed ranges. The precision you get depends heavily on granularity, meaning how many values share one scale. Per-tensor scaling is cheap but a single outlier stretches the range for everyone. Per-channel scaling gives each output row its own scale. Per-group scaling, typically 128 consecutive weights per scale, is the default for 4-bit weights because it contains outliers locally.

Floating-point formats carry their own exponent, so they tolerate wide ranges without a zero point. FP8 comes in two encodings: E4M3 (maximum 448, more mantissa, used for weights and activations in inference) and E5M2 (maximum 57,344, more range, mostly used for gradients in training). Hopper and Ada GPUs run FP8 natively on tensor cores. Blackwell adds 4-bit floating point (E2M1) with block scales in two flavours, compared below.

Weights, activations and the KV cache

There are three different things you can quantize, and they have different risk profiles.

Weights are static, so they can be quantized offline with as much calibration effort as you like. They are also the bulk of the bytes. Weight-only quantization (W4A16, W8A16) is the safest large win for decode-heavy chat and agent workloads.

Activations are produced per token at run time. LLM activations contain a few channels with magnitudes far larger than the rest, so naive per-tensor INT8 activation quantization wrecks accuracy. Dynamic per-token scales, SmoothQuant-style rebalancing, or FP8's wider range are the standard fixes. The pay-off is faster prefill and higher throughput at large batch sizes.

The KV cache grows with context length times batch size and quickly rivals the weights. For Llama 3 70B (80 layers, 8 KV heads, head dimension 128) each token stores 2 x 80 x 8 x 128 = 163,840 values, which is 320 KiB in BF16; an 8K-token sequence holds about 2.5 GiB. Storing the cache in FP8 halves that, which directly doubles how many concurrent sequences fit. Long-context retrieval tasks are the first to show damage from aggressive KV quantization, so evaluate them specifically.

Most recipes leave the embedding table and the lm_head output projection in 16-bit. They are a small share of the parameters, and the output projection directly shapes the token distribution.

How the main algorithms work

Round-to-nearest (RTN) simply rounds each group with the formula above. It needs no data and works well at 8 bits, but at 4 bits it loses noticeable quality on many models.

GPTQ quantizes a layer one column at a time and, after rounding each column, adjusts the not-yet-quantized columns to compensate for the error just introduced. The adjustment uses an approximate Hessian, H = 2 X X^T, built from the layer's calibration inputs X. That is why GPTQ needs calibration data: it minimises the error of the layer's output on realistic inputs, not the error of the weights themselves.

AWQ (activation-aware weight quantization) observes that a small fraction of weight channels matter far more because they multiply large activations. Instead of keeping those channels in 16-bit, it multiplies them by a per-channel scale before quantization and divides the matching activations by the same factor, which is mathematically equivalent in full precision but gives the salient weights more of the 4-bit grid. The scale is found by a small search on calibration data.

SmoothQuant targets W8A8. It moves the outlier problem from activations into weights with a per-input-channel factor s_j = max|X_j|^a / max|W_j|^(1-a), commonly with a = 0.5, then folds the factor into the previous layer so it costs nothing at run time. Rotation methods such as QuaRot and SpinQuant go further, multiplying by orthogonal matrices that spread outliers across all channels before quantizing.

Here is the core of group-wise symmetric quantization, the step all of these methods eventually perform:

import torch

def quantize_groupwise(w: torch.Tensor, bits: int = 4, group: int = 128):
    """w: [out_features, in_features] in fp16/bf16. Returns int codes and per-group scales."""
    out_f, in_f = w.shape
    assert in_f % group == 0
    qmax = 2 ** (bits - 1) - 1                      # 7 for int4, symmetric range [-8, 7]
    wg = w.float().reshape(out_f, in_f // group, group)
    scale = wg.abs().amax(dim=-1, keepdim=True).clamp(min=1e-8) / qmax
    q = torch.clamp(torch.round(wg / scale), -qmax - 1, qmax).to(torch.int8)
    return q.reshape(out_f, in_f), scale.squeeze(-1).half()

def dequantize(q, scale, group: int = 128):
    out_f, in_f = q.shape
    return (q.float().reshape(out_f, in_f // group, group) * scale.float().unsqueeze(-1)).reshape(out_f, in_f)

w = torch.randn(4096, 4096, dtype=torch.bfloat16)
q, s = quantize_groupwise(w)
err = (dequantize(q, s) - w.float()).pow(2).mean() / w.float().pow(2).mean()
print(f"relative MSE: {err:.4%}")   # RTN baseline; GPTQ/AWQ lower the *output* error, not this number

Kernels and memory layout: the format is a contract

A quantized checkpoint is only useful if a kernel can consume it without unpacking it into a full 16-bit copy in HBM, which would erase the memory saving. Two 4-bit codes fit in a byte, or eight in a 32-bit word, and the scales sit in a separate tensor. A fused kernel loads packed codes and scales into registers or shared memory, dequantizes there, and feeds the tensor cores in the same pass. Kernels such as Marlin pre-shuffle the packed weights at load time so that each warp's loads are contiguous and the dequantization instructions overlap with memory traffic.

This makes the format a contract between the recipe and the runtime. Group size, symmetry, whether zero points exist, the packing order, which layers were skipped, and the activation scheme are all recorded in the checkpoint's quantization config. If the runtime has no kernel for that exact combination on your GPU, it either refuses to load or falls back to a slow path. FP8 compute needs Hopper, Ada or newer; FP4 tensor-core paths need Blackwell. Check the support matrix before choosing a recipe.

Worked example: one 70B replica per GPU

Suppose you serve a Llama 3 70B-class model for an internal assistant on 80 GB GPUs and want one GPU per replica instead of two. Do the arithmetic first. About 2.1 billion parameters live in the embedding table and the untied lm_head; keep those in BF16 for about 4.2 GB. The remaining roughly 68.5 billion linear-layer weights at about 4.125 effective bits (INT4 codes plus one 16-bit scale per 128) take about 35 GB. Total: roughly 40 GB of weights. Leave a few gigabytes for the runtime, activations and CUDA graphs, and about 35 GB remains for KV cache: roughly 107,000 tokens in BF16, or roughly double that with an FP8 cache. That is the capacity you are buying.

The offline job with LLM Compressor, the quantization library maintained alongside vLLM, looks like this. Use calibration text from your own traffic where you can (scrubbed of personal data); 512 samples of 2,048 tokens is a reasonable starting point.

from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier

model_id = "meta-llama/Meta-Llama-3-70B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

# 4-bit weights, group size 128, activations stay 16-bit; keep the output head in full precision.
recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])

oneshot(
    model=model,
    dataset="my_calibration_set",       # or a Hugging Face dataset name / your own Dataset object
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=512,
)
model.save_pretrained("llama3-70b-w4a16-g128", save_compressed=True)
tokenizer.save_pretrained("llama3-70b-w4a16-g128")

Serving reads the quantization config from the checkpoint, so you mostly configure the cache and memory budget:

vllm serve ./llama3-70b-w4a16-g128 \
    --kv-cache-dtype fp8 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.92

Check the startup log for the number of KV cache blocks the runtime allocated and confirm it matches your estimate. A much smaller figure usually means a layer was not quantized as you expected.

Evaluating a quantized model

Perplexity is a weak signal: a model can keep it and still lose reasoning or code correctness. Use three layers.

1. Distribution checks. Run the same prompts through the baseline and the quantized model and measure the KL divergence of the next-token distributions and the rate at which the greedy token flips. These are cheap and sensitive.
2. Task evals. Score your real task suite plus standard benchmarks that match your use (code, maths, long-context retrieval, tool calling) against the BF16 baseline with identical decoding settings. Fix your tolerance in advance, for example no task worse than one point.
3. Performance evals. Measure time to first token, inter-token latency and throughput at your real batch-size and context-length distribution, plus the KV capacity actually allocated. A format that helps at batch 1 can lose at batch 64 if its kernel is less mature.

Failure modes

• Calibration mismatch. Calibrating a coding assistant on generic web text protects the wrong channels. Symptoms appear only in the domain you care about. Calibrate on representative prompts.
• Outlier layers. A few layers (often the first and last blocks, or MoE router layers) are far more sensitive. Per-layer KL analysis finds them; keep them in higher precision.
• Format and kernel mismatch. Loading a checkpoint on a GPU without a native kernel silently uses a slow path or fails at startup. Pin runtime versions and test on the exact target GPU.
• Long-context degradation. Aggressive KV quantization degrades retrieval deep in long prompts while short-prompt evals look perfect. Include needle-style and long-document tasks.

Choosing a format: trade-offs

What to do next

1. Measure your workload's prefill-to-decode ratio and batch sizes; pick weight-only or W8A8 from that, not from a leaderboard.
2. Compute the weight and KV memory arithmetic for your model and GPU, as in the worked example, before running anything.
3. Check your runtime's quantization support matrix for your exact GPU generation.
4. Build a calibration set of 256-512 representative prompts and run one GPTQ or AWQ W4A16 job.
5. Gate it with KL divergence, your task evals and a latency test at production concurrency.
6. Ship behind a canary, keep the BF16 replica warm for rollback, and re-run the gate on every model update.
7. Read on: AWQ versus GPTQ, FP8 quantization, KV cache quantization, the Marlin kernel and quantization evaluation methodology.