Why architecture matters here
AWQ vs GPTQ architecture matters because the quantization method affects both serving speed and quality. AWQ tends to give slightly faster inference on modern kernels because its transformation is compatible with fused matmul; GPTQ often gives slightly better quality on complex tasks. The gap is small — 0.5-1 point on MMLU — but real.
Cost matters. Quantization is a one-time cost, but the resulting artifacts have different runtime performance. A team that picks the wrong method loses 10-20% of throughput they could have had.
Reliability comes from calibration data. Domain-appropriate calibration produces better quantization; generic calibration underperforms. Both AWQ and GPTQ depend on this.
The architecture: every step explained
Walk the diagram top to bottom.
FP16 Model. The starting point. Full-precision weights.
Calibration Data. 128-512 representative samples. Both methods use this to estimate activation statistics.
Method Choice. AWQ or GPTQ (or newer variants like SmoothQuant, HQQ). Pick based on target runtime and quality requirements.
GPTQ (layer-by-layer). Process one linear layer at a time. Use the Optimal Brain Surgeon framework to round weights to INT4 while updating remaining weights to compensate. Requires computing and inverting a hessian approximation per layer. Slow to run but high quality.
AWQ (activation-aware). Identify weight channels with high activation magnitudes. Scale those channels down before quantizing (and absorb the inverse scale into the previous layer). Standard INT4 quantization on the scaled weights. Preserves precision where it matters most.
GPTQ output. INT4 weights + per-group scales. Weight matrix is straightforward.
AWQ output. INT4 weights + per-group scales. Additionally, the previous linear layer absorbed a scaling transformation; the two together produce the same math as pre-quantization.
Common format. Both produce INT4 groupwise (typically group size 128) with per-group FP16 scales. The bit layout on disk is essentially the same; the difference is in how the values were chosen.
Runtime kernels. vLLM, TensorRT-LLM, llama.cpp, and MLC each ship kernels for AWQ and GPTQ. AWQ often has better fused-kernel support; GPTQ has slightly wider hardware coverage.
Trade-off summary. AWQ: typically 5-10% faster inference on modern kernels, slightly larger quality drop on complex tasks. GPTQ: slower quantization run, smaller quality drop typically, wider ecosystem.
End-to-end quantization runs
Trace a quantization run. Choose to quantize Llama-3-70B for INT4 serving in vLLM.
You gather 128 calibration examples from your domain. You run AWQ using AutoAWQ: for each linear layer, run one forward pass on calibration data, capture activation magnitudes per input channel. Identify top ~1% channels with highest activation. Compute a scale that preserves their precision.
Absorb the scale into the previous layer's weights (multiply by inverse). Quantize the current layer's weights: INT4 groupwise with per-group FP16 scale. Save the packed weights.
Alternate: run GPTQ using AutoGPTQ. For each linear layer, compute the approximate hessian from calibration activations. Use the OBS framework: quantize one weight column at a time, updating remaining columns to minimize error. Slower — hours on a 70B model — but quality is often superior.
Both output HuggingFace-compatible models. Load into vLLM. Kernels dispatch to the appropriate quantized matmul.
Evaluate: MMLU, HumanEval, domain-specific. AWQ scores 77.9 vs baseline 79.1; GPTQ scores 78.3. On your domain benchmark, GPTQ wins by 0.4 points. Inference: AWQ delivers 32 tps; GPTQ delivers 30 tps. Depending on your priorities, pick one.