Why it matters

Quantization is often the difference between 'can't run this model' and 'runs fine on my laptop'. Understanding it opens up deployment options and cuts costs.

Advertisement

The architecture

Post-training quantization: quantize a trained FP16 model. Simple, quick, small quality loss.

Quantization-aware training: simulate quantization during training. Better quality but more expensive.

Quantization approachesPTQpost-training, fastQATaware trainingGPTQ / AWQsophisticatedGPTQ and AWQ are current standards: layer-by-layer, calibration data, minimal quality loss
Quantization spectrum.
Advertisement

How it works end to end

INT8: 2x smaller than FP16, minor quality loss. Standard for production.

INT4: 4x smaller, larger quality loss but often acceptable. Popular for on-device.

Sub-4-bit (2-bit, 1.58-bit BitNet): experimental, big quality trade-off. Some promising results.

Modern methods: GPTQ (calibration data + layer optimization), AWQ (activation-aware weight quantization), GGUF (llama.cpp format).