A neural-network weight is a number, but where that number sits in memory — next to which other numbers, on which cache line, at which address — decides how fast your model runs almost as much as how many numbers there are. A matrix is a two-dimensional idea, yet memory is a one-dimensional array of bytes, so every framework has to pick a rule for flattening rows and columns into a line of addresses. That rule is the layout, and it is the quiet protagonist of matmul performance: the same multiply can run several times faster or slower depending only on whether the inner loop walks memory in order or jumps around it. This article works from first principles — row-major versus column-major, the stride formula that turns an index into an address, why cache lines make contiguous access cheap and strided access expensive, the real cost of a transpose, tiled layouts for GEMM, alignment and padding, and how llama.cpp mmaps a GGUF file straight into the address space. Throughout, the distinction that matters most: the bytes on disk are not automatically the layout the CPU wants to compute on.
Memory is 1-D; a matrix is 2-D
A weight matrix W with shape [m, n] is a grid in our heads, but RAM is a flat, linearly addressed array of bytes. To store the grid we must choose an order in which to walk its m × n entries and lay them down one after another. There are two natural choices, and essentially every system picks one of them.
Row-major (C, NumPy default, PyTorch default) writes the whole first row, then the whole second row, and so on: entries within a row are neighbors in memory. Column-major (Fortran, MATLAB, and the classic BLAS/LAPACK convention) writes the whole first column, then the second: entries within a column are neighbors. The mathematical matrix is identical; only the byte order differs. This sounds like bookkeeping, but it is the single fact that determines whether a given loop reads memory sequentially — the pattern hardware is built to love — or leaps across it in strides. Every later topic here (cache behavior, transposes, tiling, alignment, mmap) is downstream of this one choice, so it is worth nailing the arithmetic before reasoning about speed.
Strides: turning an index into an address
The bridge from a 2-D index (i, j) to a 1-D byte address is the stride: how many elements you skip to advance one step along each axis. For a row-major [m, n] matrix of a dtype that is s bytes wide, the address is:
addr(i, j) = base + (i * n + j) * s (row-major)
stride_row = n * s stride_col = s
addr(i, j) = base + (j * m + i) * s (column-major)
stride_row = s stride_col = m * sRead the row-major formula: moving to the next column (j → j+1) advances the address by exactly one element, s bytes — the columns of a row are packed tight. Moving to the next row jumps a full n * s bytes. Column-major is the mirror image: rows are tight, columns jump. A stride of s means ‘contiguous’; a stride of thousands of bytes means ‘scattered.’ PyTorch stores these strides explicitly, which is how a .t() or .transpose() can be free: it just swaps the two stride values and leaves the bytes untouched, producing a non-contiguous view. The numbers did not move; only the map for reading them changed.
Cache lines make contiguous access cheap
Why does stride matter to speed? Because the CPU never fetches one number at a time. It fetches a cache line — 64 bytes on essentially every modern x86 and Arm core — and caches it whole. Sixty-four bytes is 16 float32 values, or 32 float16/bfloat16 values, or 64 int8 values.
When your loop reads memory contiguously, the first access to a line pays the cost of a fetch and the next 15 (or 31, or 63) elements are already sitting in L1 — nearly free. This is spatial locality, and it is the whole reason contiguous layouts win. Worse, the hardware prefetcher watches for sequential and fixed-stride patterns and pulls the next lines in before you ask, hiding memory latency entirely. Now consider a loop that walks a row-major matrix down a column: each step jumps n * s bytes, landing on a fresh cache line every time. You fetch 64 bytes to use 4, wasting 15/16 of your memory bandwidth and defeating the prefetcher. Same data, same math, but an order of magnitude more memory traffic — purely because the access pattern fought the layout instead of following it.
The matmul: which layout the inner loop wants
A linear layer computes Y = X W^T (PyTorch’s nn.Linear stores W as [out, in]). Strip it to the core: an output element is a dot product of one row of the activation with one row of the weight matrix, summed over the shared in dimension. The performance question is whether the two vectors you dot together are each contiguous, because a dot product streams straight down its inputs.
This is why nn.Linear stores the weight as [out, in] in row-major order: the in weights feeding a single output neuron are contiguous, so computing one output is a clean sequential sweep over one weight row and one activation row — perfect spatial locality on both operands. Had the weight been laid out so that the reduction axis was strided, every multiply-accumulate would touch a new cache line and the kernel would crawl. The general lesson: a matmul kernel is fast when the axis it reduces over is the contiguous axis for both operands. Layout is not decoration around the algorithm; it is chosen so that the hot inner loop reads memory in the order the hardware wants to deliver it.
A worked example: mapping a matrix to addresses
Make it concrete. Take W with shape [3, 4] (3 rows, 4 columns) of float32 (s = 4 bytes), stored row-major at base address 0x1000 (4096). The address of each element is 4096 + (i*4 + j)*4:
j=0 j=1 j=2 j=3
i=0 0x1000 0x1004 0x1008 0x100C <- row 0, contiguous
i=1 0x1010 0x1014 0x1018 0x101C <- row 1, contiguous
i=2 0x1020 0x1024 0x1028 0x102C <- row 2, contiguousWalk along a row (i fixed, j increasing) and the address climbs by 4 each step — all 48 bytes live on the single cache line starting at 0x1000 (which spans 0x1000–0x103F). One fetch serves the whole matrix. Now walk down a column (j fixed, i increasing): addresses go 0x1000 → 0x1010 → 0x1020, striding 16 bytes. On this tiny matrix they still share a line, but scale to a realistic [4096, 4096] weight: a row stride is 4096 × 4 = 16{,}384 bytes, so stepping down a column jumps 16 KB per element — a guaranteed cache miss every step. Identical data; the column walk moves ~16× more memory than the row walk. That gap, repeated billions of times, is why layout is a first-order performance concern.
Transposition is not free
If the wrong axis is contiguous, why not just transpose? Because a physical transpose — actually reordering the bytes so the other axis becomes contiguous — costs a full pass over the data with a nasty access pattern: you read one matrix contiguously while writing the other one strided (or vice versa), so one side of the copy misses cache constantly. For an [m, n] matrix that is m × n element moves plus the cache-miss tax, and it needs somewhere to put the result.
There are two escapes. First, the logical transpose already mentioned: swap the strides and change nothing in memory. That is free but only relabels the layout — the bytes are still ordered the old way, so a kernel that wanted contiguity on the new axis is no better off. Second, and this is the standard trick, high-performance BLAS never transposes at all: sgemm takes transA/transB flags and simply reads the operand in whichever order it needs, fusing the transpose into the multiply so no extra pass exists. The lesson for weights: prefer to store them in the layout the kernel will consume, so that neither a physical shuffle nor a locality-killing strided read is ever required at runtime.
Blocked and tiled layouts for GEMM
Even with the right major order, a large matmul cannot keep everything in cache. An L1 data cache is ~32–48 KB; a [4096, 4096] float32 matrix is 64 MB. Stream naively and you evict data you will need again moments later, re-reading operands from DRAM many times over. The fix is tiling (cache blocking): partition the matrices into small sub-blocks — say 64 × 64 — sized so the working set of a block-times-block multiply fits in a cache level, and finish all the arithmetic touching a block while it is resident before moving on.
Tiling can be just a loop structure over a normal matrix, but the fastest kernels go further and repack the weights into a block-contiguous layout: the elements of each tile are stored consecutively, so the kernel reads a whole tile as one sequential run and even the tile boundaries stay cache-friendly. This is why libraries like oneDNN and llama.cpp’s kernels sometimes convert a plain [out, in] matrix into an internal blocked format before the hot loop. The math is unchanged; the bytes are rearranged so that both the register tiling and the cache hierarchy are fed contiguous data at every level.
Weight packing for quantized formats
Quantized weights add a packing problem on top of layout, and the sibling article on on-disk formats covers the specifics — here is just the shape of it. Sub-byte weights (4-bit, and lower) do not align to byte boundaries, so a format must decide how to pack multiple weights into each byte and how to interleave them with their scales and zero-points. GGUF’s Q4_K-style blocks, for instance, store a run of quantized values together with the block’s scale metadata in a fixed super-block struct.
The layout goal is the same as for full-precision weights: arrange the packed nibbles so the dequantize-and-multiply kernel reads them contiguously and can unpack a whole SIMD register’s worth at once, keeping each block’s scale next to the values it rescales so there is no second strided fetch for metadata. Because the packed layout is dictated by the kernel that will consume it, quantized weights are typically the least portable: two runtimes can hold the same logical 4-bit tensor in byte layouts that are not interchangeable without a repack. The principle carries over intact — contiguity for the consumer — only now the ‘element’ is a packed block, not a single scalar.
Alignment and padding
Beyond order, where a run of data begins matters. A SIMD load of a 512-bit AVX-512 vector moves 64 bytes at once, and it is fastest — and on some instructions required — when that 64 bytes starts on a 64-byte boundary, i.e. an address divisible by 64. This is alignment. An aligned vector load touches exactly one cache line; a misaligned one can straddle two, doubling the fetch and occasionally faulting on strict instructions.
So allocators and formats align the start of each tensor, and they pad rows to keep every row aligned too. If a row’s natural length is not a multiple of the alignment, a few unused bytes are appended so the next row still starts on a boundary — which is why an in-memory row stride can exceed n × s. The cost is a sliver of wasted space and the discipline of never assuming stride_row == n * s; the payoff is that every row, every tile, every SIMD load lands cleanly. On-disk formats bake this in: GGUF pads tensor data to an alignment (32 bytes by default) so each tensor begins on an aligned offset, which becomes an aligned pointer the moment the file is mapped.
On-disk format vs in-memory layout
Here is the distinction that ties the article together, and the one most often conflated. The on-disk format is a serialization: a byte stream with a header describing each tensor’s name, shape, dtype, and offset, followed by the raw tensor bytes (SafeTensors and GGUF both take this shape). The in-memory layout is how those numbers must be arranged for the compute kernel — a specific major order, specific strides, specific alignment, possibly a tiled or repacked form.
These two are related but not the same, and the relationship decides your load cost. If the on-disk bytes already match the layout the kernel wants, loading is essentially free — point the runtime at the bytes and compute. If they differ — the kernel wants a transposed, tiled, or differently packed form — then ‘loading’ secretly includes a conversion pass that copies and rearranges every weight, costing time and a second copy in RAM. Good local-inference formats are therefore designed backwards from the kernel: they store weights in exactly the layout the runtime consumes, so that the on-disk format and the in-memory layout coincide and the expensive repack disappears.
mmap: mapping weights straight into memory
When the two layouts coincide, you unlock the technique that makes local LLM loading feel instant: memory-mapped I/O. Instead of read()-ing the weight file into a freshly allocated buffer (which copies every byte and needs RAM for the whole model up front), mmap() maps the file directly into the process’s virtual address space. The weights now have addresses, but nothing is loaded yet.
Pages are faulted in lazily by the OS the first time the kernel actually touches them, straight from the file into the page cache — no explicit copy, no user-space buffer. This is exactly how llama.cpp loads a GGUF model, and the benefits compound: start-up is near-instant because you do not wait for a multi-gigabyte read; two processes mapping the same model share the physical pages instead of each holding a copy; and under memory pressure the OS can drop clean weight pages and re-fault them from disk for free, since the file is the backing store. The precondition for all of it is that the on-disk bytes are already in a usable layout — which is precisely why GGUF stores aligned, kernel-ready tensor data. mmap is the reward for making the disk format equal the memory layout.
SIMD and vectorization want contiguity too
Cache locality is one reason to keep the reduction axis contiguous; vectorization is the other, and it reinforces the same layout. A SIMD instruction multiplies or accumulates a whole vector of lanes in one go — 16 float32 lanes for AVX-512, more for lower precisions. To feed it, the compiler wants to issue one wide, aligned load that pulls 16 consecutive weights into a register.
That is only possible when those 16 weights are consecutive in memory. If the axis being reduced is strided, the vector unit cannot do a single load; it must gather elements from scattered addresses — a slow operation that throws away most of SIMD’s advantage. So the contiguous-reduction-axis rule pays off twice: it gives spatial locality for the cache and it lets the reduction vectorize with clean loads. This is a big part of why storing nn.Linear weights as [out, in] row-major is not arbitrary — the in axis is both the reduction axis and the contiguous axis, so a single output neuron’s dot product is a stream of aligned vector loads and fused multiply-adds, which is about the best a CPU can do.
Practical implications for CPU SLMs
On a CPU running a small language model, decode is memory-bandwidth-bound: each generated token must stream the entire weight set through the cores, and the arithmetic is cheap next to the cost of moving those bytes. That makes layout decisive — the ceiling on tokens-per-second is set by how efficiently weights flow from RAM through the cache hierarchy into the vector units, and a layout-hostile access pattern can leave you reading the same lines repeatedly or wasting most of every fetch.
Three practical consequences follow. First, prefer a runtime and format whose on-disk layout matches its kernel, so mmap works and no repack tax is paid at load. Second, quantization helps layout as much as it helps footprint: 4-bit weights are one-eighth the bytes of float32, so eight times as many weights ride each cache line and each unit of bandwidth — the reason quantized models often decode faster on CPU, not merely fit in less RAM. Third, trust the library’s packed/tiled layout rather than second-guessing it; the blocked format that looks strange in a hex dump is exactly what keeps the hot loop fed. On a bandwidth-bound CPU decoder, being kind to the cache is the same thing as being fast.
Common pitfalls
A handful of layout mistakes recur, and each maps to something above. Assuming contiguity. A tensor that has been transposed, sliced, or viewed can be non-contiguous — its strides no longer equal n * s — and code that treats its data_ptr() as a dense row-major block will read garbage. Call .contiguous() when a kernel demands it, but know it is a real copy, not a free relabel.
Silent transpose-on-load. If your on-disk layout and your kernel’s layout disagree, the framework may insert a conversion pass you never see — slow start-ups and a doubled memory spike during load are the symptoms. Column-walking a row-major matrix (or vice versa) in a hand-written loop quietly costs an order of magnitude in bandwidth; swap the loop nesting so the innermost index is the contiguous one. Ignoring alignment gives correct but slower SIMD, with the occasional straddled cache line. And fighting the library’s packed layout by converting to a ‘normal’ shape for the hot path throws away the tiling the kernel depends on. Every one of these is the same root error in a new costume: an access pattern that disagrees with the byte order it is walking.
(i, j) is base + (i*n + j)*s in row-major order, so walking a row is contiguous and walking a column strides by a whole row; because the CPU fetches 64-byte cache lines and vectorizes contiguous runs, the fast kernel is the one whose reduction axis is the contiguous axis — exactly why nn.Linear stores weights as [out, in]. A physical transpose costs a strided pass, so good code stores weights in the layout the kernel consumes and lets BLAS read the rest. Tiling, alignment, and padding all serve the same master: keep every load contiguous and aligned. The distinction that ties it together is that the on-disk format and the in-memory layout are not the same thing — make them coincide, and llama.cpp can mmap a GGUF file straight into memory with no copy and no repack. Layout is not housekeeping; on a bandwidth-bound CPU decoder it is the performance.