AWQ vs GPTQ Quantization Architectures Compared: Hessian error compensation versus activation-aware scaling

The shared problem: one layer, sixteen levels

Everything happens one linear layer at a time. A layer computes Y = W X with W of shape [out, in]. Group quantization splits each row of W into runs of g input channels (128 by default) and maps each run to sixteen levels:

scale = (max(w_group) - min(w_group)) / 15
zero  = round(-min(w_group) / scale)          # asymmetric; symmetric variants fix zero = 8
q     = clamp(round(w / scale) + zero, 0, 15)  # stored as 4 bits
w_hat = scale * (q - zero)                     # what the kernel reconstructs

Round-to-nearest (RTN) applies that formula and stops. At 8 bits it is nearly lossless; at 4 bits it is often visibly worse on reasoning and long-form tasks, for two reasons. First, the error in each weight is independent, but the layer output sums thousands of them, and nothing stops the errors piling up in one direction for the inputs the model actually sees. Second, a few input channels carry activations far larger than the rest, so error in the weights attached to them is multiplied up.

Both methods instead make the layer's output close on real inputs, minimising ||W X - W_hat X||^2 over calibration activations X. GPTQ attacks the first problem directly and the second implicitly. AWQ attacks the second directly and leaves the first to RTN.

The shared problem: one layer, sixteen levels

Everything happens one linear layer at a time. A layer computes Y = W X with W of shape [out, in]. Group quantization splits each row of W into runs of g input channels (128 by default) and maps each run to sixteen levels:

scale = (max(w_group) - min(w_group)) / 15
zero  = round(-min(w_group) / scale)          # asymmetric; symmetric variants fix zero = 8
q     = clamp(round(w / scale) + zero, 0, 15)  # stored as 4 bits
w_hat = scale * (q - zero)                     # what the kernel reconstructs

Round-to-nearest (RTN) applies that formula and stops. At 8 bits it is nearly lossless; at 4 bits it is often visibly worse on reasoning and long-form tasks, for two reasons. First, the error in each weight is independent, but the layer output sums thousands of them, and nothing stops the errors piling up in one direction for the inputs the model actually sees. Second, a few input channels carry activations far larger than the rest, so error in the weights attached to them is multiplied up.

Both methods instead make the layer's output close on real inputs, minimising ||W X - W_hat X||^2 over calibration activations X. GPTQ attacks the first problem directly and the second implicitly. AWQ attacks the second directly and leaves the first to RTN.

GPTQ: compensate the error you just made

GPTQ (Frantar et al., ICLR 2023) descends from Optimal Brain Surgeon. For the squared output error, the second-derivative matrix with respect to one row of weights is H = 2 X X^T, an [in, in] matrix shared by every row of the layer. Rounding weight i introduces an error; the optimal fix is a correction to the remaining unrounded weights proportional to row i of H^-1. GPTQ's engineering contribution was making that tractable at 175B scale: quantize columns in a fixed order for all rows at once, read the needed rows of H^-1 from a single Cholesky factorisation instead of updating an inverse after each step, and batch the updates in blocks of 128 columns so the GPU does matrix multiplies rather than rank-one updates.

import numpy as np

def group_params(w, maxq=15):
    lo, hi = np.minimum(w.min(1), 0), np.maximum(w.max(1), 0)
    scale = np.maximum((hi - lo) / maxq, 1e-8)
    return scale[:, None], np.round(-lo / scale)[:, None]

def rtn(w, scale, zero, maxq=15):
    return scale * (np.clip(np.round(w / scale) + zero, 0, maxq) - zero)

def gptq_layer(W, X, group=128, damp=0.01, act_order=False):
    # W: [out, in] weights; X: [in, n_tokens] inputs captured for this layer.
    W = W.astype(np.float64).copy()
    H = 2 * X @ X.T
    perm = np.argsort(-np.diag(H)) if act_order else np.arange(W.shape[1])
    W, H = W[:, perm], H[perm][:, perm]
    H += damp * np.mean(np.diag(H)) * np.eye(len(H))   # dampening keeps H invertible
    U = np.linalg.cholesky(np.linalg.inv(H)).T          # upper factor of H^-1
    Q = np.zeros_like(W)
    for i in range(W.shape[1]):
        if i % group == 0:                               # fit on already-corrected weights
            scale, zero = group_params(W[:, i:i + group])
        Q[:, i:i+1] = rtn(W[:, i:i+1], scale, zero)
        err = (W[:, i:i+1] - Q[:, i:i+1]) / U[i, i]
        W[:, i+1:] -= err @ U[i:i+1, i+1:]               # compensate in later columns
    return Q[:, np.argsort(perm)]

Three details matter in production. Dampening adds a fraction of the mean diagonal (1% is the usual default) so H stays positive definite when some channels never fire in the calibration set. Group parameters are fitted lazily, on weights that already absorbed corrections from earlier columns. Act-order (desc_act) processes columns with the largest diag(H) first, while there are many unrounded columns left to absorb their error. It usually helps quality, but group membership then follows the permuted order, so the checkpoint must carry a g_idx tensor mapping each input channel to its group.

AWQ: move the error away from what matters

AWQ (Lin et al., MLSys 2024) starts from an observation: keeping roughly 0.1-1% of weight channels in FP16, chosen by activation magnitude rather than weight magnitude, recovers most of RTN's loss. AWQ gets that effect without mixed precision through an exact identity. For any positive per-channel vector s, W X = (W diag(s)) (diag(s)^-1 X). Scaling a salient channel's weights up before rounding shrinks their relative rounding error; the inverse scale is folded into whatever produced X, so it costs nothing at inference.

def fake_quant(W, group=128):
    out = np.empty_like(W)
    for j in range(0, W.shape[1], group):
        scale, zero = group_params(W[:, j:j + group])
        out[:, j:j + group] = rtn(W[:, j:j + group], scale, zero)
    return out

def awq_scales(W, X, n_grid=20, group=128):
    # W: [out, in]; X: [n_tokens, in] inputs to this layer (and its siblings).
    x_mag = np.abs(X).mean(0)                    # salience per input channel
    ref = X @ W.T
    best_err, best_s = np.inf, None
    for k in range(n_grid):
        s = np.clip(x_mag ** (k / n_grid), 1e-4, None)
        s = s / np.sqrt(s.max() * s.min())       # keep scales centred near 1
        W_hat = fake_quant(W * s, group) / s     # round the scaled weights, undo scale
        err = np.mean((X @ W_hat.T - ref) ** 2)
        if err < best_err:
            best_err, best_s = err, s
    return best_s                                # then: W *= s, previous op /= s

The search is a single exponent alpha in [0, 1) per group of layers, not a free vector, which is why AWQ needs little data and does not overfit the calibration set easily. Where the inverse scale goes is architecture-specific: for a Llama-style block, the RMSNorm before attention absorbs it for q/k/v, the norm before the MLP absorbs it for gate/up, up_proj absorbs it for down_proj, and v_proj can absorb it for o_proj only when the shapes line up (grouped-query attention often prevents it). A second search then shrinks each group's clipping range a little when that lowers output error. After both, the weights are rounded with plain RTN.

Side by side

Put side by side, the two passes share inputs and outputs and almost nothing in between:

GPTQ has more freedom, so with good calibration data it can match outputs more closely, and with skewed data it can fit the skew. AWQ's few parameters make it robust to calibration choice and quick to run, but it cannot recover error that salience scaling does not reach.

What lands on disk

Open a legacy GPTQ or AutoAWQ checkpoint and each quantized linear layer appears as a handful of tensors instead of one weight:

• qweight: int32 words, each packing eight 4-bit values. The packing axis and nibble order differ between formats, which is why the format name in quantization_config matters.
• scales: FP16, one per group per output channel, shape [in/g, out].
• qzeros: packed 4-bit zero points, one per group per output channel. Older GPTQ checkpoints store these with a historical off-by-one that loaders correct for, so never edit them by hand.
• g_idx: GPTQ with act-order only; int32 group index for each input channel.

Newer pipelines write a different container. The vLLM project's llm-compressor, which absorbed AutoAWQ after that repository was archived, implements both algorithms as modifiers and saves in the compressed-tensors format, so the tensor names and quant_method differ from the legacy files even though the numbers mean the same thing. Inspect what you have rather than assuming:

import json
from safetensors import safe_open

cfg = json.load(open("model/config.json"))["quantization_config"]
print(cfg.get("quant_method"), cfg.get("bits"), cfg.get("group_size"), cfg.get("desc_act"))
with safe_open("model/model.safetensors", "pt") as f:
    for k in f.keys():
        if "layers.0.mlp.down_proj" in k:
            print(k, f.get_slice(k).get_shape(), f.get_slice(k).get_dtype())

Sizes are easy to predict. A Llama-3-8B-shaped model has about 6.98B parameters in its decoder linear layers. At 4 bits plus a 16-bit scale and a 4-bit zero per 128 weights, that is about 4.16 bits per weight, or 3.63 GB. Embeddings and the LM head (2 x 128,256 x 4,096) usually stay FP16 at 2.10 GB, so the file is about 5.7 GB against 16.1 GB in FP16. Both methods land on the same number: the size depends on the format, not the algorithm.

From checkpoint to kernel

At load time the server reads quantization_config and picks a kernel. On NVIDIA GPUs of compute capability 8.0 and newer, vLLM repacks both legacy GPTQ and AWQ checkpoints into the layout of the Marlin mixed-precision kernel when the configuration is supported, and falls back to older kernels otherwise. At decode batch sizes the layer is memory-bound, so reading a quarter of the bytes is where the speed-up comes from. Details are in the Marlin kernel article.

Two consequences follow. First, serving speed is set by the kernel, not by the algorithm that chose the integers; with both checkpoints on the same kernel, throughput is usually indistinguishable, so treat claims that one method is inherently faster with suspicion. Second, act-order used to be a speed trap: older kernels had to gather scales through g_idx in the inner loop. Marlin-style kernels handle it by permuting weights once at load and the activations at run time, so the cost is small, but on other backends check before shipping desc_act=True.

Worked example: an 8B model both ways

Take a Llama-3-8B-shaped model, 32 decoder layers of seven linear projections each, quantized on one 24 GB GPU with 256 calibration sequences of 2,048 tokens. Both pipelines walk the model block by block, keeping only one block in GPU memory and caching that block's inputs, roughly 256 x 2,048 x 4,096 BF16 values or about 4.3 GB.

GPTQ's extra state is one Hessian per distinct layer input. The largest is down_proj, whose input width is 14,336, so H is 14,336 squared FP32 values, 0.82 GB, plus its Cholesky factor. The same layer in a 70B model has a 28,672-wide input and a 3.3 GB Hessian, which is why large-model GPTQ runs offload aggressively.

AWQ's extra state is tiny: one mean-magnitude vector per input. Its cost is forward passes: for each of 20 grid points it re-quantizes the layer group and runs the cached inputs through it.

Then evaluate both against FP16 using the same harness: perplexity on held-out text, KL divergence of next-token distributions, and your task suite, as described in the quantization evaluation methodology. If the two are within noise, prefer the one that was cheaper to produce and easier to reproduce.

Failure modes and combining the two

• Calibration mismatch. Code or multilingual models calibrated on English prose lose most on exactly the traffic you serve. GPTQ is more exposed because it fits more. Sample from production prompts.
• Too few tokens for the Hessian. If the calibration set has fewer tokens than down_proj has input channels, H is rank-deficient and dampening is all that keeps it invertible. Use enough sequences that tokens far exceed the widest input.
• MoE experts that never fire. Rarely routed experts receive few or zero calibration tokens, so both methods quantize them nearly blind. Check per-expert token counts and fall back to RTN or higher precision for starved experts.
• Fold targets that do not exist. AWQ needs a preceding op to absorb 1/s. New architectures need explicit mappings; a missing one silently skips scaling for that layer.
• Outlier-heavy activations. Both are weight-only methods. If you also want INT8 or FP8 activations, you need SmoothQuant or rotation methods on top.

Combining the two is legitimate: llm-compressor documents recipes that apply AWQ's scaling and then GPTQ's rounding in one pass, since the first changes magnitudes and the second chooses integers. Treat it as a third candidate in your evaluation, not a guaranteed win. For the packing arithmetic underneath all of these formats, see bit packing for quantized weights.

What to do next

1. Collect 256-512 calibration sequences from real prompts, covering every language and task you serve.
2. Produce three artifacts from the same FP16 model: RTN, AWQ and GPTQ with act-order, all at group size 128.
3. Inspect each checkpoint's quantization_config and tensor names; confirm the server picks the kernel you expect from its startup log.
4. Measure KL divergence and task accuracy against FP16 on held-out data, not the calibration set.
5. Benchmark throughput at your real batch sizes; expect a tie and investigate if there is not one.
6. Record quantizer version, recipe and calibration hash next to the artifact so it can be rebuilt.