INT8 Calibration for LLMs, in depth: choosing static activation scales, observers, calibration data, sensitivity sweeps and the evaluation gate

What calibration chooses

In symmetric INT8 quantization, a tensor is represented by integers from -127 to 127 and one floating-point scale. Every value is divided by the scale, rounded, clamped to the integer range, and multiplied back when needed. Choosing the scale is the same as choosing a clipping threshold: the largest magnitude that will be represented without saturation.

# Symmetric INT8, one scale per tensor
#   s = clip / 127
#   q = clamp(round(x / s), -127, 127)
#   x_hat = q * s
# Error has two parts:
#   rounding error for |x| <= clip: uniform, at most s/2 per value
#   clipping error for |x| >  clip: |x| - clip
# Calibration = choosing 'clip' to minimise the total error that matters downstream.

The two error terms pull in opposite directions. A large clip covers every value but makes the step size coarse, so small values, which are the majority, lose precision. A small clip gives fine steps but saturates the largest values. Calibration means picking the clip that minimises the error that matters, ideally measured at the output of the layer or the model rather than on the tensor itself.

Static calibration is not the only option. Dynamic quantization computes the scale per token at runtime from the token's actual maximum, which needs no calibration data and tracks every input exactly, at the cost of a reduction over each row before every matrix multiply and a scale that cannot be folded into the preceding operation. Many serving stacks use dynamic per-token activation scales for this reason. Static scales remain attractive where every operation must be fused into integer kernels, on accelerators without fast reductions, and for the KV cache, so calibration is still worth understanding. The trade-offs among per-tensor, per-channel and per-token are covered in quantization granularity.

Why LLM activations defeat the naive method

In transformer LLMs above a few billion parameters, a small number of hidden dimensions carry activations far larger than the rest, consistently across tokens and layers. The LLM.int8() work identified these emergent outlier features, and SmoothQuant built on the observation that the outliers live in fixed channels. A per-tensor scale set by min-max is dominated by those few channels.

A small worked example shows the effect. Suppose one input to a projection has most values within plus or minus 2, while one channel reaches 60. Min-max calibration sets the clip to 60, so the scale is 60/127, about 0.47. Every value between -0.24 and 0.24 rounds to zero, and every value from about 0.71 to 1.18 lands on the same integer, 2. Clipping at 4 instead gives a scale of about 0.031 and fine resolution for the bulk of the values, but now the outlier channel is saturated from 60 to 4, and because that channel carries meaningful signal, the layer output changes badly. Neither scale works, which is why a single per-tensor scale for a raw LLM activation is often unusable without help.

The help comes in three forms. Smoothing divides each input channel by a factor and multiplies the corresponding weight rows by the same factor, migrating the difficulty into the weights, which are quantized per channel and tolerate it. Rotation methods apply an orthogonal transform that spreads outliers across all channels. Or the stubborn layers stay in higher precision. In practice you combine smoothing with careful calibration and a short fallback list.

The worst offenders are well known. The input to the MLP down projection, after the gated activation, has the heaviest tails in many architectures. Attention outputs and the residual stream near the first and last layers are also frequent problems. Calibration and the sensitivity sweep should treat these as suspects from the start.

Building the calibration set

The calibration set determines what the scales are fitted to, so it should look like production. Common practice is a few hundred sequences, often 128 to 512, of 512 to 2048 tokens, which is enough because each sequence contributes thousands of token activations per layer. More data rarely helps once the distribution is right; the wrong distribution is the real risk.

• Use the deployment format. If the model serves chat, render calibration samples with the same chat template and system prompt. Template tokens produce their own activation patterns.
• Cover the traffic mix. Code, other languages, structured output and tool calls each produce different activation ranges. A set drawn only from generic web text under-represents them.
• Include long contexts if you serve them, because activation statistics at later positions and in attention differ from short prompts.
• Exclude the evaluation data. Calibrating on the test set inflates the results you will use to decide.

Observers compared

An observer watches activations during calibration and turns them into a clip. Four families are common.

For LLMs the best results usually come from measuring error where it matters: at the layer output, the block output or the model loss. A practical recipe is to start from MSE search, check per-layer output error, and compare against min-max on the layers that have been smoothed, where min-max often becomes good enough. The code below collects statistics with forward pre-hooks on every linear layer's input and supports three methods.

import torch, torch.nn as nn

class Observer:
    def __init__(self):
        self.absmax = 0.0
        self.buf = torch.empty(0)             # capped subsample of |x|
    def update(self, x, cap=1 << 20):
        a = x.detach().abs().float().flatten()
        self.absmax = max(self.absmax, a.max().item())
        idx = torch.randint(0, a.numel(), (min(a.numel(), 65536),), device=a.device)
        self.buf = torch.cat([self.buf, a[idx].cpu()])
        if self.buf.numel() > cap:            # bound memory per layer
            self.buf = self.buf[torch.randperm(self.buf.numel())[:cap]]

    def clip(self, method="mse"):
        a = self.buf
        if method == "minmax":
            return self.absmax
        if method == "p99.99":
            return torch.quantile(a, 0.9999).item()
        # MSE search over candidate clips
        best, best_err = self.absmax, float("inf")
        for frac in torch.linspace(0.05, 1.0, 96):
            c = frac.item() * self.absmax
            s = c / 127
            xh = torch.clamp(torch.round(a / s), -127, 127) * s
            err = torch.mean((xh - a) ** 2).item()
            if err < best_err:
                best, best_err = c, err
        return best

def attach(model):
    obs, handles = {}, []
    for name, m in model.named_modules():
        if isinstance(m, nn.Linear) and "lm_head" not in name:
            obs[name] = Observer()
            handles.append(m.register_forward_pre_hook(
                lambda mod, inp, n=name: obs[n].update(inp[0])))
    return obs, handles

@torch.no_grad()
def calibrate(model, tokenizer, prompts, max_len=2048, method="mse"):
    obs, handles = attach(model)
    for text in prompts:                      # already rendered with the chat template
        ids = tokenizer(text, return_tensors="pt", truncation=True,
                        max_length=max_len).input_ids.to(model.device)
        model(ids)
    for h in handles:
        h.remove()
    return {n: o.clip(method) / 127 for n, o in obs.items()}   # scale per Linear input

This is a teaching implementation: it keeps a random subsample per layer to bound memory and treats each linear input as one tensor. A production tool fuses the scales into kernels, shares one scale across the query, key and value projections when they read the same input, and stores scales alongside the checkpoint.

The sensitivity sweep

Even with good scales, a few layers lose disproportionately. Find them by quantizing one layer input at a time, leaving the rest in full precision, and measuring the increase in held-out loss. Sort by the increase and keep the top few in FP16 or BF16; the cost in speed is usually small because only a handful of layers are affected.

@torch.no_grad()
def sensitivity(model, scales, heldout, base_loss):
    # Fake-quantize one Linear input at a time; report loss increase.
    report = []
    for name, s in scales.items():
        mod = model.get_submodule(name)
        h = mod.register_forward_pre_hook(
            lambda m, inp, s=s: (torch.clamp(torch.round(inp[0] / s), -127, 127) * s,))
        delta = eval_loss(model, heldout) - base_loss
        h.remove()
        report.append((delta, name))
    return sorted(report, reverse=True)       # worst first; keep the top few in FP16

Repeat the sweep with all other layers quantized as well, because errors interact; a layer that is harmless alone can tip the model when everything else is already noisy.

The KV cache

The KV cache is a separate calibration target. Storing keys and values in INT8 halves cache memory compared with FP16, which directly raises the batch size or context length a server can hold. Keys often have outlier channels like other activations, so per-channel or per-head key scales tend to work better than one scale per layer, while values are usually better behaved. Calibrate cache scales with long sequences, because statistics drift with position, and evaluate on long-context tasks specifically. See KV-cache quantization for the formats and kernels.

The evaluation gate

A calibrated model ships only if it passes a gate defined before calibration started. Measure held-out perplexity on data drawn like the calibration set but disjoint from it, accuracy on the task evaluations that matter to the product, and long-context retrieval if you serve long prompts. Add token agreement: the share of positions where the quantized model's top-1 token matches the full-precision model on the same prompts. Agreement is cheap, sensitive, and catches problems that averages hide. Set thresholds relative to the full-precision model and to run-to-run noise rather than absolute numbers, and record them with the checkpoint so the next calibration is judged the same way.

Failure modes

• Wrong calibration distribution. Scales fitted to web prose clip code or non-English activations; quality drops only for those users.
• Missing chat template. The model degrades on the first tokens of every response because template positions were never seen.
• Stale scales after a fine-tune. A checkpoint fine-tuned or merged with an adapter keeps the old scales; recalibrate after any weight change.
• Averages hide damage. Perplexity moves little while a specific task, often arithmetic or structured output, breaks. Gate on task evaluations and agreement.
• Short-sequence calibration. The KV cache or late-position activations saturate on long prompts.
• Scale mismatch at fusion. Fused projections that share an input but were calibrated separately end up with inconsistent scales and silent errors.

Trade-offs and alternatives

Static W8A8 gives the fastest integer kernels and the simplest graph once scales are fixed, at the price of calibration effort and sensitivity to distribution shift. Dynamic per-token activation scales cost a little runtime and remove most of that sensitivity. Weight-only INT8 or INT4 needs no activation calibration at all and suits memory-bound decoding, but keeps matrix multiplies in higher precision. On GPUs with FP8 tensor cores, FP8 is often easier, because its exponent covers a much wider range and outliers saturate less, though it still needs scales.

What to do next

1. Decide static versus dynamic per-token activation scales for your serving stack before investing in calibration.
2. Build a calibration set of a few hundred production-like sequences rendered with your chat template, disjoint from your evaluation data.
3. Apply smoothing, then calibrate with MSE search and compare against min-max on smoothed layers.
4. Run a one-layer-at-a-time sensitivity sweep, keep the worst layers in FP16 or BF16, and repeat with everything quantized.
5. Calibrate KV-cache scales on long sequences, with per-channel or per-head key scales.
6. Gate on held-out perplexity, task evaluations, long-context tests and top-1 agreement, and recalibrate after every weight change.