SmoothQuant — Redistributing Quantization Difficulty

Why it matters

W8A8 gives maximum hardware speedup. SmoothQuant is a leading technique to make it work at quality. Understanding matters for latency-critical inference.

The architecture

Problem: activations have outliers making INT8 range poor.

Solution: pre-multiply activations by 1/s (small), post-multiply weights by s (large). Preserves matmul result but shifts precision requirements.

How it works end to end

s selection: per-channel scale factor. Calibrated from activation distributions. Migration strength α (0-1) controls tradeoff.

Preserves matmul: W · x = (W · s) · (x / s). Same result, different quantization difficulty.

Combines with GPTQ / AWQ for further improvement.