Quantization Layouts in Memory

Why layout, not just bit-width, is the story

The headline of quantization is ‘int4’ or ‘int8,’ but that number alone predicts neither the file size nor the speed. Two int4 schemes can differ by 20% in actual bytes and by a lot more in accuracy, purely because of how the values are grouped and what metadata rides alongside them. Layout is where the abstraction meets the hardware.

There are two reasons layout dominates. First, low-bit weights need scales to be reconstructed, and those scales are not free — how many you store, and at what precision, is a real fraction of the total footprint at 4 bits and below. Second, decode on a CPU is memory-bandwidth-bound: each generated token must stream the whole weight matrix from RAM, so total bytes moved is the clock. Halving the bytes nearly halves the time — but only if the dequantization work you add to unpack those bytes stays cheap enough to hide under the memory latency. Both of those pressures — the scale overhead and the unpack cost — are layout questions, not bit-width questions. Get the layout wrong and a ‘4-bit’ model is neither small nor fast.

The affine map: floats to integers and back

The workhorse of practical quantization is the affine (uniform) map. A real weight x is encoded as an integer q using a positive scale (a float) and an integer zero_point:

quantize:    q = clamp( round(x / scale) + zero_point,  q_min,  q_max )
dequantize:  x_hat = scale * (q - zero_point)

The two lines are inverses up to rounding. scale sets the width of each quantization bucket (the real-world distance between two adjacent integers), and zero_point is the integer that maps back to real zero — it lets an asymmetric range (say all-positive activations) use the full integer span. The round introduces quantization error of at most half a bucket, ± scale/2, and the clamp introduces clipping error for any value outside [q_min, q_max]. Everything downstream — symmetric vs asymmetric, per-tensor vs per-group — is just a different policy for choosing scale and zero_point and for how many of them to keep. The map itself never changes.

Choosing scale and zero-point from the range

Given a set of weights with observed minimum x_min and maximum x_max, and a target integer range [q_min, q_max] (for unsigned 4-bit that is [0, 15]), the affine parameters fall straight out of matching the endpoints:

scale       = (x_max - x_min) / (q_max - q_min)
zero_point  = round( q_min - x_min / scale )

This ‘min-max’ calibration guarantees no clipping — both extremes land exactly on an integer — but it is fragile: a single outlier stretches the range, inflating scale so that all the ordinary weights crowd into a few buckets and lose resolution. That single fact drives most of the field. You can clip deliberately (shrink the range past the true max, trading a little clipping error for much finer buckets on the bulk of the distribution), or you can refuse to let one scale cover values that do not belong together — which is exactly the motivation for finer granularity. The zero-point rounding also matters: it must land on a representable integer so that real zero encodes exactly, which keeps padding and pruned weights clean.

Symmetric vs asymmetric

The first layout fork is whether zero_point is forced to zero. In symmetric quantization the range is centered on zero, [-a, +a], and zero_point = 0, so dequant collapses to a single multiply x_hat = scale * q. In asymmetric quantization the range is [x_min, x_max] with a non-zero zero_point, costing an extra subtract per weight but fitting lopsided distributions without waste.

The trade is precision-per-bit against metadata and arithmetic. Weights are often roughly zero-centered and near-symmetric, so symmetric is the common choice for them — it needs no stored zero-point at all, saving both bytes and the inner-loop subtract. Activations, by contrast, are frequently one-sided (think post-ReLU, or the all-positive tail of a GELU), so wasting half the integer range on values that never occur is expensive; asymmetric earns its extra zero-point there. In layout terms the difference is concrete: a symmetric block stores just packed integers plus a scale; an asymmetric block additionally stores a zero-point (or an equivalent min), and those extra bits count against the effective bits-per-weight we tally later.

Granularity: per-tensor, per-channel, per-group

The second fork — the one that most defines a layout — is how many scales you keep. Three levels dominate. Per-tensor: one scale (and one zero-point) for the entire weight matrix. Cheapest metadata, but a single outlier anywhere ruins resolution everywhere. Per-channel: one scale per output channel — for a weight matrix W[out, in], one scale per row. Each output neuron gets its own range, which is a large accuracy win for essentially free.

Per-group (a.k.a. block or group-wise): partition each row along the input / reduction dimension into contiguous groups of g weights (typical g = 32, 64, or 128) and give each group its own scale (and zero-point, if asymmetric). This is the sweet spot for sub-4-bit weight quantization, and the grouping axis is deliberate: because the matmul accumulates a dot product along the input dimension, striding through groups is exactly striding through the reduction — so the right scale is always the one for the block you are currently multiplying. Smaller g means finer adaptation to local outliers and better accuracy, at the cost of storing more scales. That tension is the entire bits-per-weight budget, and we make it numeric below.

Packing sub-byte values: two nibbles to a byte

Memory is addressed in bytes, but an int4 weight is 4 bits, so you must pack two weights into every byte. The standard layout puts one weight in the low nibble and the next in the high nibble:

byte = (q_hi << 4) | (q_lo & 0x0F);   // pack two int4 into one byte
q_lo = byte & 0x0F;                  // unpack low nibble
q_hi = (byte >> 4) & 0x0F;           // unpack high nibble

The same idea generalizes to any sub-byte width, just less tidily: int3 and int5 do not divide 8, so real formats either pad to a convenient boundary or pack a fixed number of weights into a fixed number of bytes (for example, 8 three-bit weights straddle 3 bytes). This is why block sizes are powers of two and why odd bit-widths carry awkward packing rules. Two consequences follow for layout. First, weights are no longer individually addressable — you read a byte and split it. Second, the order in which the nibbles are packed is a free parameter, and formats exploit it: interleaving the packing so that a single vector instruction produces a run of already-in-order dequantized values is a real speed lever, discussed shortly.

The unpacking tax in the inner loop

Packing shrinks the file, but every weight now costs a little arithmetic to recover before it can be used. For each int4 value the dequant path is a mask, maybe a shift, a subtract of the zero-point, and a multiply by the scale:

q      = (byte >> shift) & 0x0F;      // extract nibble
x_hat  = scale * (float)(q - zero_point);  // dequantize

Done scalar-per-weight in a hot matmul, that overhead would swamp the savings. The rescue is SIMD: a single AVX2 or NEON instruction unpacks and dequantizes 8, 16, or 32 nibbles at once, so the amortized cost per weight is a fraction of an instruction. And the crucial context is that decode is memory-bound — the core is usually waiting on RAM for the next block of weights anyway, so the unpack arithmetic slots into cycles that would otherwise be idle. Dequantization is often effectively free precisely because the bottleneck is bandwidth, not compute. That is the whole bargain: you spend a few cheap ALU ops to move far fewer bytes, and on a bandwidth-starved CPU that trade is a large net win.

Scales live next to their block

A block is useless without its scale, so the layout stores them together. A typical 4-bit block is a small struct: the packed quantized values, immediately followed (or preceded) by the scale and, if asymmetric, the zero-point or min. In llama.cpp’s GGUF format the classic Q4_0 block is literally 32 four-bit weights plus one fp16 scale, laid out contiguously.

This co-location is deliberate and it is a performance decision, not just tidiness. Because decode streams weights linearly and the dot product walks the reduction dimension group by group, keeping each scale adjacent to the block it rescales means the scale arrives in the same cache line, already in L1 when the inner loop needs it — no separate pointer chase into a distant scale array, no extra cache miss per block. The alternative, a struct-of-arrays layout with all quantized values in one region and all scales in another, is friendlier for some bulk operations but forces two memory streams in the hot loop. For bandwidth-bound decode, the interleaved ‘array-of-blocks’ layout — values and their scale in one contiguous record — is the one that wins, which is why the mainstream CPU formats all use it.

Effective bits per weight: counting the metadata

Once scales ride alongside the data, ‘4-bit’ is a lie of omission. The honest number is effective bits per weight: all the bits in a block divided by the weights it holds. For a symmetric int4 block of g = 32 weights with one fp16 (16-bit) scale:

bits = 32 weights x 4 bits  +  16-bit scale
     = 128 + 16  =  144 bits  over 32 weights
eff  = 144 / 32  =  4.5 bits / weight   (0.5 bit, ~12.5%, overhead)

Make it asymmetric by adding an fp16 zero-point (or min) and it becomes (128 + 16 + 16) / 32 = 5.0 bits/weight. Grow the group to g = 128 and the symmetric case drops to (128×4 + 16) / 128 = 4.125 bits/weight — the scale is amortized over four times as many weights, so overhead falls from 12.5% to about 3%. There is the core tension in one line: smaller groups buy accuracy and cost bits; larger groups save bits and cost accuracy. This is also why serious formats quantize the metadata too — storing scales at 6 or 8 bits instead of 16 — to claw back overhead without coarsening the grouping.

A worked memory example: a 7B model on a laptop

Turn the bits-per-weight into gigabytes. Take a 7-billion-parameter model and price the weights at several layouts (using GB = 10^9 bytes; GiB is ~7% smaller):

This is the number that decides whether a model fits in a laptop’s RAM, and — because decode is bandwidth-bound — it is also roughly proportional to tokens per second. The jump from 14 GB to under 4 GB is what lets a 7B model run at all on a machine with 8 GB of RAM, and the ~10% spread within the int4 row (3.61 vs 4.38 GB) is entirely a layout choice: group size, and whether you paid for a zero-point. ‘4-bit’ alone did not tell you any of that.

Anatomy of one block

It helps to picture a single 4-bit block as it actually sits in memory: a run of packed nibbles, then the small scalar metadata that reconstitutes them. The diagram shows the symmetric Q4_0-style record — 32 four-bit weights (16 bytes) plus a 2-byte fp16 scale — and the effective-bits arithmetic it implies.

Layout built for a fast dequant-and-multiply

The reason these layouts look the way they do is a single hot loop: weight-only quantization keeps activations in fp16/fp32 and stores only the weights low-bit, so the matmul inner loop must dequantize a block of weights and immediately multiply-accumulate it against a float activation vector. The layout is engineered so that step is one smooth SIMD sweep: load a packed block, load its adjacent scale, unpack the nibbles into a vector register, convert to float, and fuse-multiply-add against the activations — no gathers, no branches, no scattered scale lookups.

This is why the nibble packing order is chosen so that one unpack instruction yields values already in the order the dot product wants them. llama.cpp’s int4 types, for instance, interleave the two halves of each block (weights 0–15 in the low nibbles, 16–31 in the high nibbles) precisely so a single mask-and-shift produces two aligned sub-vectors. The layout is not the storage format that happened to be convenient — it is reverse-engineered from the SIMD kernel that will read it. Every decision (block size a power of two, scale in the same cache line, interleaved nibbles) exists to keep the dequant-and-multiply loop branch-free and bandwidth-limited rather than compute-limited.

GGUF k-quants: hierarchical scales

The formats you actually download embody these ideas. In the GGUF ecosystem (llama.cpp), the simplest types are legacy quants: Q4_0 (32-weight blocks, symmetric, one fp16 scale — 4.5 bpw) and Q4_1 (adds an fp16 min for asymmetry — 5.0 bpw). The modern k-quants (Q4_K, Q5_K, Q6_K, and friends) add a level of hierarchy to cut metadata overhead.

A k-quant works on a super-block of 256 weights split into eight sub-blocks of 32. Each sub-block still gets its own scale and min for local adaptation, but those per-sub-block scales are themselves quantized to just 6 bits and rescaled by one fp16 super-block scale. The payoff is visible in the accounting: Q4_K lands at about 4.5 bits/weight (256×4 data bits, plus 8×6 bits of scales and the same of mins, plus two fp16 super-scalars, over 256 weights) — the same effective size as crude Q4_0, but with 32-weight granularity instead of one coarse scale, so materially better accuracy for the same footprint. The mixed _K_M/_K_S variants push this further by spending more bits on the layers that matter most.

GPTQ and AWQ: smarter rounding, same layout family

GGUF k-quants mostly decide the layout; GPTQ and AWQ are post-training methods that decide the rounding, then store the result in a familiar packed-int4-plus-group-scales layout. Both are weight-only, typically 4-bit, group-wise (commonly g = 128), and both are aimed at GPU inference but share this article’s memory structure.

GPTQ is second-order: layer by layer, it uses (approximate) Hessian information to choose each weight’s rounding direction so as to minimize the error in that layer’s output, not just the per-weight rounding error — and it updates the not-yet-quantized weights to compensate as it goes. AWQ (activation-aware weight quantization) observes that a small fraction of weight channels are salient because they multiply large-magnitude activations; it scales those channels up before quantizing (and folds the inverse scale into the layout) so the important weights keep resolution while the range stays tame. The lesson for layout: the on-disk structure — packed low-bit integers with per-group scales — is remarkably shared, and what distinguishes these methods is how cleverly they pick the integers that go into that structure.

Pitfalls and CPU-SLM implications

Several layout traps recur. Outliers are the big one: a coarse (per-tensor) scale sacrifices the whole distribution to a few extreme values, which is why per-group is the floor for good sub-4-bit weights — and why activation quantization is harder than weight quantization, since activation outliers are systematic (the problem SmoothQuant and AWQ target). Metadata precision is another: store scales too coarsely and you reintroduce error through the back door; store them too finely and your ‘4-bit’ model quietly weighs five. And group size is a genuine accuracy/size dial, not a default to ignore — 32 vs 128 is a real decision.

For CPU-hosted small language models the implications are direct. Decode is bandwidth-bound, so the effective bits-per-weight is very nearly your speed budget, and shaving overhead (larger groups, quantized scales) buys tokens per second. The dequant work is usually free because the core is waiting on memory — but only if the kernel is SIMD and branch-free, which is exactly what the interleaved, scale-adjacent block layout is built to enable. The practical advice: pick a format by its effective bits-per-weight and its accuracy at that size (k-quants and GPTQ/AWQ at 4-bit are the sweet spot for 7–8B models on a laptop), and trust that the layout underneath was already reverse-engineered from the inner loop that has to read it.