Why it matters

INT8 is the workhorse of production LLM inference. Understanding its mechanics and quality trade-offs is essential for deployment.

Advertisement

The architecture

Weight quantization: for each tensor (or per-row), find min and max. Compute scale = (max - min) / 255. Store as int8 + scale.

Activation quantization: same but computed dynamically or via calibration.

INT8 model layoutWeightsint8 + scaleActivationsint8 dynamicMatmulint8 accelerationSymmetric vs asymmetric; per-tensor vs per-channel scales — trade complexity for quality
INT8 quantization.
Advertisement

How it works end to end

Symmetric: scale only, no zero point. Simpler.

Asymmetric: scale + zero point. Better precision for asymmetric distributions.

Per-tensor vs per-channel: per-channel scales handle intra-tensor variation better.

Hardware: NVIDIA Tensor Cores + CPU AVX-512 VNNI accelerate INT8 matmul.