Quantization Overview — Trading Precision for Speed and Size

Why it matters

Quantization is often the difference between 'runs' and 'doesn't run' for LLMs on modest hardware. Understanding the precision options and their trade-offs is core deployment knowledge.

The architecture

Precision spectrum: FP32 (full), FP16/BF16 (half), INT8, INT4, sub-4-bit (experimental).

Each halving roughly halves memory + accelerates on supporting hardware.

How it works end to end

Post-training quantization: quantize a trained model. Simple, quick.

Quantization-aware training: simulate quantization during training. Better quality but expensive.

Weight-only vs weight+activation: weight-only is safer (smaller quality loss); activation quantization is faster.