Why architecture matters here
The architecture matters because on-device constraints are unforgiving and interlocking. RAM is scarce, so the weights must be quantized; startup must be instant, so the weights cannot be copied; the hardware is heterogeneous, so the same file must run on an x86 laptop, an ARM phone, an Apple GPU, and a mobile NPU. A naive 'load the tensors into a float array and call a matrix library' approach fails on every one of these axes at once. GGUF and its runtime are what you get when you optimize the entire path — file layout, loading, dequantization, and kernel dispatch — for those constraints together rather than one at a time.
It matters because quantization is the enabling trick and the format is built around it. A 7-billion-parameter model in 16-bit floats is roughly fourteen gigabytes, which will not fit in a phone. The same model quantized to four bits per weight is under four gigabytes, which will. GGUF stores tensors in block-quantized formats — a small group of weights shares a scale (and sometimes a minimum), so each weight costs four or five bits instead of sixteen. The runtime's job is to hold those compact blocks in memory and expand them to floating point only transiently, inside the matmul, so the footprint reflects the quantized size while accuracy stays acceptable.
It matters because memory mapping is not merely a startup optimization; it changes the memory model of the whole process. Because the weights are a read-only mapping of a file, the kernel can evict them under pressure and page them back from disk, and several inference processes on the same box share a single physical copy. That is why a local runtime can open a multi-gigabyte model in the blink of an eye and why running two chats against the same model does not double the memory. The trade is that the first pass over a cold model pays page faults, which is why real deployments often 'warm' the model or lock its pages.
Finally, it matters because the backend abstraction is what lets one file and one runtime span wildly different hardware. The compute graph is defined once in terms of tensor operations, and a backend layer maps those operations onto CPU SIMD instructions, a CUDA or Metal GPU, or a vendor NPU. You can even split a model — keep some layers on the CPU and offload others to the GPU when accelerator memory is limited. That portability is a large part of why the ecosystem coalesced on GGUF: author the model once, and it runs, quantized, on almost anything with a few gigabytes of memory.
The architecture: every piece explained
The GGUF file itself is the foundation: a magic header, a version, a table of key-value metadata, and then the tensor data. The metadata names the architecture (how many layers, attention heads, the embedding dimension), the hyperparameters, and the tokenizer vocabulary and merge rules — everything needed to reconstruct the model without any external file. The tensors follow, each tagged with its shape and its quantization type, and laid out so they can be addressed directly by offset once the file is mapped.
Memory mapping is the loading strategy. Instead of reading bytes into a heap buffer, the runtime calls mmap and receives a pointer to the file's contents in its address space. No weight is actually in RAM yet; the operating system pages each one in the first time a kernel touches it. This makes 'loading' a multi-gigabyte model nearly free and lets the OS manage the weights as reclaimable page cache shared across processes. The runtime then parses the metadata to learn the model's shape and quantization scheme.
With the shape known, the runtime builds a compute graph: the sequence of embeddings, attention blocks, feed-forward layers, and normalization that defines a forward pass, each node bound to a backend kernel. Alongside it allocates the KV cache, a buffer sized by the number of layers, heads, and the maximum context length, which will hold the key and value vectors for every token so attention does not recompute the past. The KV cache is frequently the largest single runtime allocation and the thing that decides how long a context you can afford.
During the forward pass, each quantized weight block is dequantized on the fly — read from the mapped file, expanded from its four- or five-bit representation back to fp16 using its stored scale, and consumed immediately by a matmul, never materialized as a full float tensor in memory. The backend kernels do the actual arithmetic: hand-tuned SIMD on the CPU, CUDA on an NVIDIA GPU, Metal on Apple silicon, or a mobile NPU path. The dispatch is chosen per operation, so a model can run partly on CPU and partly on GPU.
Inference then splits into two phases. Prefill processes the entire prompt in one batched pass, filling the KV cache for all prompt tokens with large, efficient matmuls. The decode loop follows, generating one token at a time: each step attends over the growing KV cache, produces a distribution, samples a token, appends its key and value to the cache, and repeats. Prefill is compute-bound and fast per token; decode is memory-bandwidth-bound because it reads the whole KV cache and all weights for a single token, which is why local generation speed is dominated by how fast memory can feed the cores. The ops strip names the knobs that balance all of this against the hardware you actually have.
End-to-end flow
Trace a local chat from cold. The user opens the app, which calls into the runtime with a path to a quantized .gguf. The runtime opens and mmaps the file — an operation that returns almost instantly regardless of the file's size, because nothing has been read yet — and parses the metadata block to learn that this is, say, a decoder-only transformer with a known layer count, head count, and a four-bit block-quantized weight format, along with its tokenizer.
From that metadata the runtime constructs the compute graph and allocates the KV cache for the configured context length. If the user requested GPU offload, it decides how many layers' weights to copy onto the accelerator based on available VRAM, leaving the rest to run on the CPU from the mapped file. The model is now 'loaded' in the sense that it is ready to run, though most of its weights are still only promises the OS will fulfill on first touch.
The user submits a prompt. The tokenizer, carried inside the file, turns the text into token ids. Prefill runs: the runtime pushes all prompt tokens through the graph in a batched forward pass. As each layer executes, its quantized weight blocks are paged in from the mapped file (paying a one-time fault), dequantized in registers, and multiplied against the activations; the key and value vectors for every prompt token are written into the KV cache. Because prefill batches all tokens, it saturates the cores and completes the prompt in a handful of large matmuls.
Now the decode loop begins. The runtime takes the last position's output, applies the sampling strategy — temperature, top-k, top-p — and picks the next token. That token is fed back in as the sole input for the next step: the graph runs once more, but this time attention reads the entire KV cache built so far and appends one new key/value pair. Each step therefore reads all the weights and the whole growing cache to produce exactly one token, which is why throughput is set by memory bandwidth rather than raw compute. The loop repeats until the model emits a stop token or hits the length limit.
Throughout, memory behaves gracefully because the weights are a shared, reclaimable mapping. If a second chat opens against the same file, it maps the same pages and adds no weight memory, only its own KV cache. If the OS needs RAM, it can drop clean weight pages and fault them back later. The visible cost of all this efficiency is the cold-start page-fault tax on the first generation and the memory-bandwidth ceiling on decode speed — both of which the operational knobs are designed to manage.