What's inside

Model weights (any quantization). Tokenizer vocab + merges. Metadata (context length, architecture, chat template).

Advertisement

Quantization variants

Q4_K_M (medium), Q5_K_M, Q6_K, Q8_0. Higher = better quality + larger file. K_M variants optimized mixed.

Advertisement

Conversion

llama.cpp convert.py: HuggingFace → GGUF. Then quantize.exe → smaller quants. Or use TheBloke's pre-quantized.