Why architecture matters here
The problem GGUF solves is deployment reality at the edge of the compute distribution. A 70B-parameter model in FP16 is 140GB — beyond any consumer device. At Q4_K_M (~4.7 bits effective) it is ~40GB and runs on a 48GB Mac Studio; an 8B model at Q4 fits in 5GB and runs on a phone. Quantization is not an optimization here; it is the difference between 'runs locally' and 'doesn't exist locally'. And local existence matters for reasons that survive scrutiny: privacy (medical, legal, personal data never leaves the device), latency (no network), cost (no per-token billing), and resilience (offline). GGUF is the packaging that makes this a download-and-run experience instead of a research project.
The block-quantization architecture matters because where you spend bits determines what breaks. Naive uniform quantization of a whole tensor lets one outlier weight stretch the representable range and crush the precision of everything else. Small blocks with per-block scales contain the damage to 32–256 neighbors; super-block hierarchies amortize the scale overhead that small blocks create; and importance-aware variants go further — the imatrix (activation statistics from a calibration corpus) tells the quantizer which weights the model actually exercises, so the scarce bits protect them. The result is an empirical quality ladder every local-LLM user learns: Q8 ≈ indistinguishable, Q6_K ≈ near-lossless, Q4_K_M ≈ the sweet spot, Q3 and below ≈ increasingly visible degradation, with i-quants pushing the floor lower than k-quants at equal size.
The mmap decision matters just as much operationally. Because the file layout matches memory layout, loading is page-cache-driven: the OS pages tensors in on first touch, multiple processes share one copy, and a 40GB model 'loads' in the time it takes to read metadata. It also means the format is the interoperability surface — Ollama, LM Studio, llamafile, and every llama.cpp descendant consume the same artifact, which is why 'is there a GGUF?' became the first question asked of every model release.
The architecture: every piece explained
Top row: the container. A GGUF file opens with magic bytes and version, then a key-value metadata section — architecture name, layer/head counts, RoPE scaling, tokenizer vocab and merges, chat template, quantization version — typed and extensible, so a runtime needs no sidecar configs. The tensor index follows: every tensor's name, shape, quantization type, and byte offset, with data aligned for direct SIMD/GPU access. Quant blocks are the payload: for Q4_0, blocks of 32 weights sharing one FP16 scale with 4-bit values; for k-quants, 256-weight super-blocks whose eight 32-weight sub-blocks carry 6-bit quantized scales/mins under one FP16 super-scale — bits spent on scale hierarchy buy back accuracy that flat schemes lose. mmap loading ties it together: the file is mapped read-only; pages fault in on first use; nothing is copied or transformed at load.
Middle row: the type system in practice. The k-quant family ships in S/M/L mixes: the letter is a per-tensor recipe — Q4_K_M keeps attention.wv and ffn_down at Q6_K (the empirically sensitive tensors) while the bulk sits at Q4_K; embeddings and the output head often stay higher still. I-quants (IQ2_XXS through IQ3_M) use codebook-style representations and lean on an importance matrix — activation statistics gathered by running a calibration text through the full-precision model — to weight the quantization objective; below ~3 bits, imatrix is the difference between 'compressed' and 'lobotomized'. Mixed precision per tensor is thus the norm, not the exception: a GGUF is really a per-tensor precision map. Dequant kernels execute it: on CPU, AVX2/AVX-512 and NEON paths unpack blocks and fuse dequantization into the matmul; on GPU (CUDA/Metal/Vulkan), dedicated kernels do the same per offloaded layer.
Bottom rows: runtime memory strategy. Layer offload (n_gpu_layers) splits the model: as many transformer layers as fit go to VRAM, the rest run on CPU — the knob that lets a 24GB GPU accelerate a 40GB model. The KV cache — activation memory that grows with context — can itself be quantized (q8_0, q4_0 keys/values), often the deciding factor for long contexts on small VRAM. The ops strip is the user-facing craft: pick the largest quant that fits with context headroom, verify with perplexity or task evals rather than vibes, and budget weights + KV + compute buffers against real RAM/VRAM.
End-to-end flow
Follow a model from release to token. A lab ships a 8B instruct model in safetensors FP16 (~16GB). A packager runs llama.cpp's conversion: safetensors → GGUF F16 (metadata assembled from the HF config and tokenizer), then llama-quantize produces the ladder — Q8_0 (8.5GB), Q6_K (6.6GB), Q4_K_M (4.9GB), and, with an imatrix computed from a few hundred KB of calibration text, IQ3_M (3.7GB) and IQ2_M (2.9GB). Each variant is one self-contained file uploaded alongside perplexity deltas.
A user with a 16GB MacBook downloads Q4_K_M. Load: mmap, metadata parsed in milliseconds, tokenizer built, Metal backend claims all 32 layers plus the output head into unified memory. First prompt: tokens embed, and for each layer the Metal kernels unpack Q4_K/Q6_K blocks inline with the matmuls — dequantization never materializes a full FP16 tensor; it happens in registers per tile. Prefill of a 2,000-token prompt is compute-bound and takes ~1.5s; generation is memory-bandwidth-bound — the entire 4.9GB of weights streams through the SoC's 100GB/s+ fabric once per token — yielding ~20 tokens/s. The KV cache at q8_0 keeps a 16k context inside another ~1.3GB.
Same model, different machine: a desktop with a 12GB RTX 4070 and 64GB RAM runs the 70B sibling at IQ3_M (~28GB). n_gpu_layers=28 puts 28 of 80 layers in VRAM; the rest execute on AVX-512 cores from page cache. Throughput is ~3 tokens/s — usable for chat, and impossible under any non-quantized regime. The user validates their choice the right way: runs the same eval prompts against Q4_K_M-on-cloud and IQ3_M-local, sees acceptable drift for their summarization task, and pins the file's SHA in their tooling.
The failure path teaches the ladder's lower bound: trying IQ2_XXS (19GB) to fit more layers on GPU, they watch the model's structured-output reliability collapse — JSON keys misspelled, instructions half-followed. Sub-3-bit quants of mid-sized models degrade nonlinearly, and no runtime flag fixes what the bits no longer encode. They return to IQ3_M and buy the tokens/s with a smaller context instead.