When a small language model runs slowly on a CPU, the cause is almost never that the arithmetic units are too weak. It is that the numbers cannot get to them fast enough. A modern core can issue tens of billions of multiply-adds per second, but the weights those multiplies need live in DRAM, and DRAM is a hundred times slower to reach than the registers the math actually happens in. The CPU papers over that gap with a hierarchy of caches — small, fast memories that stage data closer to the core — and how well your workload fits that hierarchy is the single biggest lever on inference latency. This piece builds the hierarchy from the ground up: the latency and bandwidth of each level, how memory moves in 64-byte cache lines, why the two kinds of locality decide hit rates, why single-stream LLM decode is fundamentally memory-bandwidth-bound, what arithmetic intensity and the roofline model tell you before you write a line of code, how cache blocking rescues matmul, and why quantization is really a bandwidth optimization wearing a numerical-precision costume.
The gap the whole hierarchy exists to hide
Every performance question on a CPU eventually reduces to one uncomfortable fact: compute got fast, memory did not keep up. A core running at 3 GHz completes a clock cycle every 0.33 nanoseconds, and with SIMD it can finish dozens of floating-point operations in that time. But a load from main memory takes on the order of 80–100 ns — roughly 250 to 300 cycles during which the arithmetic units, absent other work, sit idle. That ratio is often called the memory wall.
If a program truly touched DRAM for every operand, a 100 GFLOP/s core would deliver a small fraction of a percent of its peak. The cache hierarchy exists to make that not happen. Between the registers and DRAM sit two or three levels of SRAM cache — L1, L2, and L3 — each larger and slower than the one above it. The bet is statistical: if the data a program touches next is usually data it touched recently, or data sitting right beside something it touched recently, then most accesses are served from a fast cache and the hundred-cycle trip to DRAM becomes the rare case rather than the common one. Inference performance is decided by how often that bet pays off.
The five levels, with real latency and bandwidth numbers
Here is a representative desktop or laptop x86 core (figures vary by microarchitecture, but the orders of magnitude are what matter). Assume a 3 GHz clock, so 1 cycle ≈ 0.33 ns.
| Level | Latency (cycles) | Latency (ns) | Typical size | Approx. bandwidth |
|---|---|---|---|---|
| Registers | < 1 | ~0.3 | ~2 KB (32 × 64 B) | ~10 TB/s |
| L1 data | 4–5 | ~1.3 | 32–48 KB / core | ~1–3 TB/s |
| L2 | 12–15 | ~4–5 | 256 KB–2 MB / core | ~400–900 GB/s |
| L3 (shared) | 40–50 | ~15–20 | 8–64 MB | ~200–400 GB/s |
| DRAM | 200–350 | ~80–100 | 8–128 GB | ~50–100 GB/s |
Read the table as a cliff, not a slope. Each step down costs roughly 3–4× more latency and gives up a large chunk of bandwidth, and the final step to DRAM is the steepest: an L1 hit is ~1 ns, a DRAM miss ~80 ns, a factor of sixty or more. The size column explains the tension — the fast levels are tiny. A 3.8B-parameter model at 4-bit weighs ~2 GB, which is hundreds of times larger than the entire L3. The model simply cannot live in cache; the only question is how efficiently it streams through it.
Cache lines: memory moves in 64-byte chunks
The hardware never transfers a single byte or a single float from DRAM. It moves a whole cache line — almost universally 64 bytes on x86 and ARM — as the atomic unit. Ask for one 4-byte float and the cache controller fetches the aligned 64-byte block containing it, filling the line and handing you your four bytes.
This single design choice drives most cache behaviour. A 64-byte line holds 16 FP32 values, 32 FP16 values, or 128 INT4 weights, so a program that walks memory sequentially gets those neighbours essentially for free: one miss (the ~80 ns trip) amortizes across the whole line. A program that hops around with a large stride pays the full miss for every access and uses only 4 of every 64 bytes it dragged across the bus — a 16× waste of the scarcest resource in the machine. It also creates false sharing: two threads writing different variables that happen to land on the same line will bounce that line between their private caches as if they were fighting over one value. For inference, the practical rule is that weight matrices should be laid out and traversed so the innermost loop marches contiguously along cache lines, never against them.
Temporal and spatial locality: the two bets caches make
Caches work because real programs are not random. They exhibit two kinds of locality, and every cache mechanism targets one of them. Temporal locality is reuse in time: if you touched an address, you are likely to touch it again soon. Least-recently-used replacement is the cache betting on temporal locality — it evicts what you have not touched in a while on the theory you will not need it. Spatial locality is reuse in space: if you touched an address, you are likely to touch its neighbours soon. The 64-byte line and hardware prefetchers are the cache betting on spatial locality.
Transformer inference has a lopsided locality profile that explains everything downstream. The activations — the vectors flowing between layers — are small and hot, reused across the operations of a layer, so they enjoy strong temporal locality and mostly live in L1/L2. The weights are the opposite: each weight is read, multiplied once, and not touched again until the next token. They have excellent spatial locality (they stream contiguously) but almost no temporal locality at batch size one. A cache cannot help you reuse data you never reuse, which is the seed of the entire bandwidth-bound story.
Why single-stream decode is memory-bandwidth-bound
Autoregressive generation runs one token at a time. To produce a single token, the model must read every weight exactly once and do a small amount of arithmetic with each. There is no reuse to exploit: a weight is loaded, used in one multiply-add, and discarded. So the time to generate a token is bounded below not by how fast the core computes, but by how fast the weights can be streamed out of DRAM.
Concretely, the lower bound is simply time_per_token ≥ weight_bytes / memory_bandwidth. No cache changes this, because the working set (the whole model) is hundreds of times larger than the largest cache — every byte genuinely comes from DRAM. This is the defining fact of CPU SLM inference at batch = 1: it is a memory-bandwidth-bound workload, not a compute-bound one. It is the mirror image of prefill, where the whole prompt is processed at once and the same weights are reused across many token positions, turning the matrix-vector products into matrix-matrix products with real reuse and making prefill compute-bound. Understanding which regime you are in tells you which knob — bandwidth or FLOP/s — will actually move the needle.
A worked bound: milliseconds per token from first principles
Put numbers on it. Take Phi-3-mini (3.8B parameters) quantized to 4-bit, so the weights occupy roughly 2 GB, on a laptop with ~70 GB/s of usable memory bandwidth.
weight_bytes = 2.0 GB (3.8B params × 0.5 byte/param at INT4)
memory_bandwidth = 70 GB/s
t_token (lower bound) = 2.0 GB / 70 GB/s
= 0.0286 s
≈ 28.6 ms per token
throughput ceiling = 1 / 0.0286 s ≈ 35 tokens/secondThat ~28 ms is a floor, not a prediction. Real measured rates for this model land around 30–50 ms per token, because the KV cache adds bytes to read, the bandwidth is never 100% utilized, and there is per-token overhead. But the floor is the point: no cleverness in the kernel, no smarter instruction scheduling, no additional cores beyond what saturates the bus can push you below weight_bytes / bandwidth at batch one. When you see a decode speed roughly matching this ratio, the CPU is doing its job and the bottleneck is physics; the only ways down are fewer bytes (quantization) or a wider bus (faster RAM).
Arithmetic intensity: FLOPs per byte
To reason about bandwidth versus compute without guessing, use one number: arithmetic intensity (AI), the ratio of useful floating-point operations to bytes moved from memory.
arithmetic_intensity = FLOPs performed / bytes read from DRAM (units: FLOP/byte)AI is a property of an algorithm on a data layout, not of the hardware. It answers: for every byte I am forced to drag across the memory bus, how much math do I get to do? A low AI means you spend most of your time waiting on memory (bandwidth-bound); a high AI means you keep the arithmetic units busy (compute-bound). The decode story above is exactly a statement about AI. A matrix-vector product y = W x with W of shape [n, k] does 2·n·k FLOPs (a multiply and an add per weight) while reading n·k weights of b bytes each. So its intensity is 2·n·k / (b·n·k) = 2/b FLOP/byte — a tiny constant that does not even grow with the matrix size. For FP16 (b = 2) that is 1 FLOP/byte; for INT4 (b = 0.5) it is 4 FLOP/byte. Either way, very low.
The roofline model: one picture that predicts the bottleneck
The roofline model turns arithmetic intensity into a prediction. Plot attainable performance (FLOP/s) against arithmetic intensity (FLOP/byte). Two ceilings bound what any kernel can reach: a slanted roof set by bandwidth, and a flat roof set by peak compute.
attainable_FLOPs_per_s = min( peak_compute,
memory_bandwidth × arithmetic_intensity )Below a certain intensity you are on the slanted part — performance rises linearly with AI because you are bandwidth-limited. Above it you are on the flat part — adding intensity buys nothing because you have hit the compute ceiling. The crossover is the ridge point, and it equals peak_compute / memory_bandwidth, sometimes called the machine's balance. For a CPU that can sustain, say, 300 GFLOP/s over 70 GB/s, the ridge point is 300 / 70 ≈ 4.3 FLOP/byte. Any kernel whose intensity sits left of ~4.3 is memory-bound on this machine; anything to the right is compute-bound. This is why the model is so useful: you compute an algorithm's AI, compare it to one hardware number, and you know your bottleneck before writing the kernel.
Worked example: decode versus prefill on the roofline
Place the two inference phases on that roofline with the ridge point at ~4.3 FLOP/byte.
DECODE (batch = 1): y = W x GEMV, one token
FLOPs = 2·n·k , bytes = b·n·k
AI = 2/b = 1 FLOP/byte (FP16) or 4 FLOP/byte (INT4)
1 and 4 < 4.3 ridge ==> MEMORY-BOUND
PREFILL (N tokens): Y = X W GEMM, N rows at once
FLOPs = 2·N·n·k , bytes = b·n·k (weights read once, reused across N rows)
AI = 2·N/b
N = 512, FP16: AI = 512 FLOP/byte >> 4.3 ridge ==> COMPUTE-BOUNDThe only thing that changed is N, the number of token positions sharing the same weight read. Decode processes one position, so each weight earns just 2/b FLOPs before it is thrown away — deep in bandwidth territory. Prefill (or batched serving) reuses each weight across many rows, multiplying the intensity by N and vaulting over the ridge into compute territory. This is the precise, quantitative reason the same model on the same CPU feels compute-bound while ingesting a prompt and memory-bound while typing a reply — and why batching requests is the standard cure for decode inefficiency: it raises N, and therefore AI, at the cost of latency per request.
Cache blocking: turning a bandwidth problem into a cache problem
Prefill and training lean on large matrix-matrix products, and a naive triple loop for C = A · B squanders them. Compute each output element by streaming a full row of A and a full column of B, and for matrices larger than the cache you re-read B from DRAM once per row of A — O(N) passes over the same data. The arithmetic is fine; the memory traffic is catastrophic.
Cache blocking (also called tiling) fixes this by restructuring the loops so the work is done on small sub-blocks that fit in cache. Instead of multiplying whole matrices, you multiply a b × b tile of A by a b × b tile of B into a tile of C, keeping all three tiles resident in a fast cache level and reusing every loaded byte across the whole tile before evicting it. A tile multiply performs 2·b^3 FLOPs while touching only 3·b^2 elements, so its arithmetic intensity is ≈ b — the block size directly sets the intensity. Choose b large enough that the tiles saturate the compute roof, small enough that they fit in cache, and you have converted a bandwidth-bound sprawl into a compute-bound kernel.
Sizing the tiles: fitting three blocks in cache
The block size is not a guess; it falls out of the cache capacity. You need three tiles resident simultaneously — the A tile, the B tile, and the C accumulator tile — so the constraint is 3 · b^2 · bytes_per_element ≤ cache_size. Solve for a 1 MB L2 with FP32 data:
3 · b^2 · 4 bytes ≤ 1,048,576 bytes
b^2 ≤ 87,381
b ≤ √87,381 ≈ 295So a block of about 256 × 256 comfortably fits an L2 tile with headroom for the accumulator and other live data. In practice, real GEMM kernels (OpenBLAS, MKL, the kernels inside llama.cpp) use a hierarchy of block sizes — small register tiles that live in the vector registers, mid-size tiles for L1, larger panels for L2 and L3 — nesting the loops so each cache level is reused before the next one down is touched. They also leave margin below the raw capacity because the cache is set-associative: fill it to the brim and conflict misses evict live tile data early. The takeaway is that good matmul performance is a cache-fitting problem first and an arithmetic problem second.
How quantization helps: fewer bytes on the wire
Return to the batch-one bound, t_token ≥ weight_bytes / bandwidth. The bandwidth is fixed by the hardware, so the only variable you control is weight_bytes — and quantization is the lever that shrinks it. Storing each weight in fewer bits moves proportionally fewer bytes per token, and because decode is bandwidth-bound, the speedup is very nearly linear in the byte reduction.
# Phi-3 (3.8B), 70 GB/s bandwidth, per-token lower bound = bytes / 70 GB/s
FP32: 15.2 GB -> ~217 ms/token (4 bytes/param)
FP16: 7.6 GB -> ~109 ms/token (2 bytes/param)
INT8: 3.8 GB -> ~54 ms/token (1 byte/param)
INT4: 1.9 GB -> ~27 ms/token (0.5 byte/param)Halve the bytes, halve the floor; INT4 is ~4× faster than FP16 purely because there is 4× less data to read. Notice what quantization is not doing here: the arithmetic units were never the constraint, so making the math cheaper is a side benefit. It is a bandwidth optimization wearing a numerical-precision costume. It also raises arithmetic intensity (recall decode AI = 2/b, which rises as b falls), nudging the workload rightward on the roofline — though for single-stream decode it stays firmly memory-bound.
Prefetching, and the pitfalls that wreck locality
Modern cores fight the memory wall with hardware prefetchers that detect sequential or simple strided access patterns and pull the next cache lines in before the code asks for them, hiding DRAM latency behind useful work. This is why streaming weights contiguously is so cheap: the prefetcher stays one step ahead and the core rarely stalls. It is also why several classic mistakes are so expensive.
The pitfalls all amount to defeating locality. Large-stride or random access blinds the prefetcher and wastes 60 of every 64 bytes fetched. Row-major versus column-major mismatches turn a contiguous walk into a strided one — iterating a row-major matrix down its columns touches a new cache line every element. Conflict misses arise because caches are set-associative: addresses that are large powers of two apart map to the same set, so a power-of-two matrix stride can evict data you still need even when the cache is nominally big enough (padding the leading dimension fixes it). And the TLB is a cache too — touching many scattered pages thrashes address translation. For inference the lesson is uniform: lay out weights and activations so the hot loops march straight through memory.
Putting it together for CPU SLM inference
Assemble the pieces into a mental model you can act on. First, identify the regime: single-stream decode is bandwidth-bound (AI ~1–4 FLOP/byte, left of the ridge point), while prefill and batched serving are compute-bound (AI scales with the number of tokens sharing each weight). Optimize the one you are actually in — buying a faster core does nothing for a bandwidth-bound decode.
Second, attack the binding resource. For decode, that means moving fewer bytes: quantize the weights (INT4/INT8), keep the KV cache compact, and prefer memory with more bandwidth (dual-channel, faster DDR5). For prefill and GEMM, that means feeding the compute roof: cache-blocked kernels, SIMD-friendly contiguous layouts, and block sizes tuned to L1/L2. Third, respect the cache line and the prefetcher: contiguous, predictable access turns the 80 ns DRAM latency into a rare, well-hidden event instead of a per-element tax. None of this beats the fundamental floor — weight_bytes / bandwidth at batch one is a wall — but almost every real SLM setup runs well above that floor, and closing the gap is entirely a matter of feeding the hierarchy the way it wants to be fed.