Why architecture matters here
SmoothQuant fails on wrong migration alpha (quality drops) or missing kernel (throughput lost). Architecture matters because calibration + kernel + eval decide the outcome.
The architecture: every piece explained
The top strip is the algorithm. Activation outliers per channel identified. Smoothing factor shifts them to weight side. Weights scaled compensate. INT8 quantize both.
The middle row is tuning. Migration alpha controls shift. Calibration from activations. Kernels — int8 matmul with fast dequant. Eval ppl + task.
The lower rows are ops. Serving impact — throughput + memory. vs GPTQ/AWQ — activation quant too. Ops — checkpoint + kernel + eval.
End-to-end flow
End-to-end: SmoothQuant on 30B model. Alpha=0.5. Calibration on 512 samples. INT8 weights + INT8 activations. Kernel: int8 GEMM. Eval: ppl delta 0.05. Throughput 2x fp16 on capable hardware.