SmoothQuant, in depth: choosing alpha per layer, measuring the error it removes, and the methods that grew out of it

The transform, restated briefly

A linear layer computes Y = XW, with X of shape tokens by input channels. For any positive vector s with one entry per input channel, Y = (X diag(s)^-1)(diag(s) W). Dividing activation channel j by s_j and multiplying weight row j by s_j leaves the output exactly unchanged in real arithmetic. SmoothQuant chooses s_j = max|X_j|^alpha / max|W_j|^1-alpha, so channels with large activation outliers are divided down hard and their weights grow to compensate.

The division of X costs nothing at inference because it is folded into whatever produced X: the gain of the RMSNorm or LayerNorm in front of the attention or MLP projections, or the output rows of the previous linear layer. That folding constraint defines smoothing groups: every linear layer that reads the same tensor must use the same s. In a Llama-style block the groups are q, k and v behind the input norm; gate and up behind the post-attention norm; down behind up_proj, because the down input is silu(gate) times up and scaling up's output channel scales the product; and o_proj behind v_proj, with a caveat covered under failure modes.

Why one alpha is a compromise, and what to minimise instead

The best split depends on how uneven each side is, and that varies with depth and site. Some groups see a handful of extreme channels; the down-projection input, a product of two activations, can have a heavier tail; some groups have no outliers at all. A global alpha is tolerable everywhere rather than good anywhere.

The target to minimise is the error in the output of the group's linear layers with both operands quantized the way the serving kernel quantizes them. Activation error alone would pick alpha = 1 and wreck the weights. The code below quantizes X/s per token and W·s per output channel to INT8; if your deployment uses static per-tensor activation scales, change fq_rows to match, or the search will choose alphas that are too timid.

Step 1: collect statistics with forward hooks

One pass over calibration data gives each group the running per-channel maximum of |X|, for the formula, and a random sample of input rows, for measuring output error. One hook on the first linear of each group is enough because the group shares its input. Use a few hundred sequences that resemble production traffic.

import torch
from collections import defaultdict

def llama_groups(model):
    """Linears that read the same input must share one smoothing vector."""
    groups = {}
    for i, layer in enumerate(model.model.layers):
        a, m = layer.self_attn, layer.mlp
        groups[f"{i}.qkv"] = [a.q_proj, a.k_proj, a.v_proj]   # input: input_layernorm
        groups[f"{i}.gate_up"] = [m.gate_proj, m.up_proj]     # input: post_attention_layernorm
        groups[f"{i}.down"] = [m.down_proj]                   # input: silu(gate) * up
    return groups

@torch.no_grad()
def collect(model, groups, batches, rows_per_batch=256, max_rows=4096):
    act_max, samples, handles = {}, defaultdict(list), []
    for name, linears in groups.items():
        def hook(mod, inp, out, name=name):
            x = inp[0].detach().reshape(-1, inp[0].shape[-1]).float()
            m = x.abs().amax(dim=0)
            act_max[name] = m if name not in act_max else torch.maximum(act_max[name], m)
            if sum(t.shape[0] for t in samples[name]) < max_rows:
                idx = torch.randperm(x.shape[0], device=x.device)[:rows_per_batch]
                samples[name].append(x[idx].cpu())
        handles.append(linears[0].register_forward_hook(hook))
    for batch in batches:
        model(**batch)
    for h in handles:
        h.remove()
    return act_max, {k: torch.cat(v) for k, v in samples.items()}

Step 2: sweep alpha per group

def fq_rows(t, bits=8):
    """Symmetric fake quant with one scale per row: per token for X, per output channel for W."""
    qmax = 2 ** (bits - 1) - 1
    s = t.abs().amax(dim=-1, keepdim=True).clamp(min=1e-8) / qmax
    return (t / s).round().clamp(-qmax, qmax) * s

@torch.no_grad()
def group_error(x, linears, s):
    xq = fq_rows(x / s)
    num = den = 0.0
    for lin in linears:
        w = lin.weight.float()                 # [out, in]
        y_ref = x @ w.T
        y_q = xq @ fq_rows(w * s).T           # s scales input channels (columns)
        num += (y_q - y_ref).pow(2).sum().item()
        den += y_ref.pow(2).sum().item()
    return (num / den) ** 0.5

@torch.no_grad()
def search_group(x, linears, act_max, grid):
    w_max = torch.stack([l.weight.float().abs().amax(dim=0) for l in linears]).amax(dim=0)
    a_max = act_max.clamp(min=1e-5)
    errs = {a: group_error(x, linears, (a_max.pow(a) / w_max.clamp(min=1e-5).pow(1 - a)).clamp(min=1e-5))
            for a in grid}
    base = group_error(x, linears, torch.ones_like(a_max))
    best = min(errs, key=errs.get)
    return best, errs[best], base

def search_all(model, groups, act_max, samples, device="cuda"):
    grid = [round(0.05 * i, 2) for i in range(4, 17)]      # 0.20 ... 0.80
    report = {}
    for name, linears in groups.items():
        x = samples[name].to(device)
        report[name] = search_group(x, linears, act_max[name].to(device), grid)
    return report

Two details are easy to get wrong. The weight maximum must be taken across all linears in the group, because they share one s. And both maxima need a floor: a dead activation channel with maximum zero would otherwise produce s = 0 and a division by zero in the fold. The returned tuple holds the best alpha, its error and the unsmoothed baseline error, which is the number that tells you whether smoothing helped at all.

Each group sees full-precision inputs, ignoring upstream quantization error; a sequential variant that collects inputs from the already-quantized model matches deployment better but costs a pass per layer.

Step 3: fold the winning scales

@torch.no_grad()
def fold(norm_or_prev, linears, s, prev_is_linear=False):
    s = s.to(linears[0].weight.dtype)
    if prev_is_linear:                        # up_proj -> down_proj: scale up_proj's output rows
        norm_or_prev.weight.div_(s[:, None])
        if norm_or_prev.bias is not None:
            norm_or_prev.bias.div_(s)
    else:                                     # RMSNorm or LayerNorm in front of the group
        norm_or_prev.weight.div_(s)
        if getattr(norm_or_prev, "bias", None) is not None:   # LayerNorm has a beta too
            norm_or_prev.bias.div_(s)
    for lin in linears:
        lin.weight.mul_(s)                    # broadcast over the input dimension

Compute s in float32 from the chosen alpha, then fold. For q, k and v the producer is layer.input_layernorm; for gate and up it is layer.post_attention_layernorm; for down it is layer.mlp.up_proj with prev_is_linear=True. Before quantizing, run one batch through the original and folded models in float32: logits must match to rounding level, or a fold hit the wrong tensor or missed a bias.

A worked example: what the split buys

Take one token with an outlier channel at 60 while every other channel stays within plus or minus 2, and suppose all weight rows have a maximum of 0.05. With per-token INT8 the step is 60/127, about 0.47, so an ordinary channel spanning plus or minus 2 gets only about 4 quantization levels on each side. The weights, being flat, get the full 127.

Now smooth with alpha = 0.5. For the outlier channel s = sqrt(60)/sqrt(0.05), about 34.6, so its activation maximum drops to about 1.73 and its weight maximum rises to the same 1.73. For an ordinary channel s = sqrt(2)/sqrt(0.05), about 6.3, so its activation maximum becomes about 0.32. The token's step is now 1.73/127, about 0.0136, and an ordinary channel gets about 23 levels. On the weight side, every output channel's scale is now set by the outlier row at 1.73, so ordinary weight rows at 0.32 also get about 23 levels.

Before smoothing the levels were 4 for activations and 127 for weights; after, about 23 and 23. The product, roughly 530, is unchanged, which is the invariant. Smoothing moves precision to where it was missing; the sweep finds the split with the least total error, which need not be the even one.

Reading the search results

Print the report as a table of group name, best alpha, best error and baseline error, sorted by depth, and look for four patterns.

• Large gains at moderate alpha. The baseline error is several times the best error and the best alpha sits between roughly 0.4 and 0.7. This is SmoothQuant working as intended.
• Best alpha at the edge of the grid. Widen the grid before concluding anything. A group that wants 0.8 or more usually has extreme activation outliers; one that wants 0.2 or less may be dominated by uneven weights.
• Smoothing barely helps. The best error is close to the baseline. Either the group had no outliers, so it is fine to quantize directly, or the outliers are scattered across tokens rather than concentrated in channels, which a per-channel scale cannot fix.
• Error stays high at every alpha. These groups are candidates to keep in 16-bit. Excluding a few such groups often recovers most of the remaining accuracy at a small cost in speed, and the down-projection groups are common candidates. Check your own model rather than assuming.

Local error is not model quality, so the final judge is held-out perplexity and tasks, compared with the 16-bit model and with a single global alpha. For building that calibration and evaluation discipline, see INT8 Calibration for LLMs.

Tooling that does this for you

You rarely need to ship the hand-written search. In llm-compressor, SmoothQuantModifier takes smoothing_strength (default 0.5), optional mappings that pair each norm or linear with the layers it feeds (inferred from the architecture if omitted), and an ignore list; it clamps scales at a small minimum. It applies one strength everywhere, so the search above remains useful for choosing it and for building the ignore list.

Intel Neural Compressor searches per layer: alpha set to "auto" with auto_alpha_args such as alpha_min, alpha_max and alpha_step; its documentation suggests 0.3 to 0.7 in steps of 0.05. Record the chosen alphas and calibration data with the checkpoint.

The family that grew from smoothing

SmoothQuant's core idea, an exactly invertible transform that redistributes quantization difficulty before rounding, became a building block. The methods differ in the transform and in how its parameters are chosen.

The progression is from a diagonal scale chosen in closed form, to searched and then learned diagonal transforms, to full rotations that spread an outlier across every channel instead of shrinking it in one. Rotations handle outliers that a diagonal scale cannot, at the cost of extra transforms in the graph. They are covered in rotation-based quantization.

Failure modes

Trade-offs

Local, per-group search is fast but blind to how errors compound through depth; block-wise learned methods capture that interaction at far higher compute cost. Keeping groups in 16-bit recovers accuracy at the cost of mixed kernels and throughput. Every extra transform must be supported exactly by your serving stack, so check what your engine can load first. For how quantized activations flow through real GEMM kernels, see Activation Quantization Architecture.

What to do next

1. Collect per-channel activation maxima and token samples for every smoothing group of your model using the hook code above.
2. Make the fake-quant function match your serving kernel's activation and weight granularity.
3. Sweep alpha from 0.2 to 0.8 per group and print best and baseline error side by side.
4. Fold the chosen scales and verify float32 logits are unchanged before quantizing.
5. Evaluate held-out perplexity and tasks against the 16-bit model and a single global alpha; keep per-group alphas only if they win.
6. Mark groups where no alpha helps, keep them in 16-bit, and remeasure.