Why architecture matters here
Weight quantization loses quality when salient channels get squashed. AWQ observes that a small percentage of channels carry most of the activation magnitude; scaling them up before quantization preserves them.
The architecture matters because the calibration + scale search decide quality; the kernel decides throughput.
The architecture: every piece explained
The top strip is the AWQ algorithm. Pretrained weights in fp16. Calibration profiles activations. Salient channels identified from activation magnitudes. Scale search finds per-channel scales that preserve salient magnitudes.
The middle row is the quantization. Quantize weights to int4 with scales. Group size (128/64) trades quality vs metadata overhead. No activation quant — activations stay fp16 (weight-only quantization). Serving kernel executes int4 GEMM with fast dequantization.
The lower rows are validation. Eval harness tests perplexity + tasks. Comparison vs GPTQ + RTN. Ops handles checkpoint format + kernel matrix.
End-to-end flow
End-to-end: a team quantizes a 70B model with AWQ. Calibration on 128 samples. Salient channels detected; scales searched. Weights quantized int4 group=128. Kernel: Marlin int4 GEMM. Eval: perplexity delta 0.03, MMLU delta 0.2 pp. Memory 35 GB vs 140 GB fp16. Comparison shows AWQ beats naive RTN by 8 points on MMLU and matches GPTQ at ~half the calibration compute.