Why it matters

INT4 is what makes 70B models runnable on 24GB consumer GPUs. Understanding it opens up local LLM deployment.

Advertisement

The architecture

Storage: 2 weights per byte. Scale factor per group of weights.

Group size: 32-128 weights per scale typical. Smaller groups = better quality but more overhead.

INT4 quantization2 weights/byte8x smaller than FP32Group scales32-128 per scaleGPTQ / AWQsmart methodsNaive INT4 hurts quality; GPTQ / AWQ use calibration to minimize loss
INT4 mechanics.
Advertisement

How it works end to end

Naive INT4: quantize each weight independently. Poor quality.

GPTQ: calibration data + layer-by-layer optimization. Choose scales to minimize error.

AWQ (Activation-aware Weight Quantization): identify important weights, protect them. Even better quality.

GGUF: llama.cpp format, mixed precision (some layers INT4, others INT8, some higher).