Why it matters

GGUF is the format you'll encounter on HuggingFace for quantized models. Understanding the naming conventions helps pick the right variant.

Advertisement

The architecture

Naming: quantization type (Q for quant, K for k-quants), bit count (2, 3, 4, 5, 6, 8), size variant (_XS, _S, _M, _L for extra small to large).

Common: Q4_K_M is 4-bit k-quant, medium. Q8_0 is 8-bit, no groupings.

GGUF variantsWeight precisionQ4/Q5/Q8GroupingK-quants smartSize variantXS/S/M/LQ4_K_M is popular default: good quality-size trade-off for most tasks
GGUF naming.
Advertisement

How it works end to end

K-quants: mixed precision within a group. More precision for important weights within each group.

File format: header + metadata (model architecture, tokenizer) + weights.

llama.cpp support: any GGUF loads. Also supported by many wrappers (Ollama, LM Studio, KoboldCpp).

Trade-offs: smaller quants (Q2) fast + tiny but low quality. Larger (Q6, Q8) better quality but bigger.