CPU Cache Hierarchy and Transformer Inference

The gap the whole hierarchy exists to hide

Every performance question on a CPU eventually reduces to one uncomfortable fact: compute got fast, memory did not keep up. A core running at 3 GHz completes a clock cycle every 0.33 nanoseconds, and with SIMD it can finish dozens of floating-point operations in that time. But a load from main memory takes on the order of 80–100 ns — roughly 250 to 300 cycles during which the arithmetic units, absent other work, sit idle. That ratio is often called the memory wall.

If a program truly touched DRAM for every operand, a 100 GFLOP/s core would deliver a small fraction of a percent of its peak. The cache hierarchy exists to make that not happen. Between the registers and DRAM sit two or three levels of SRAM cache — L1, L2, and L3 — each larger and slower than the one above it. The bet is statistical: if the data a program touches next is usually data it touched recently, or data sitting right beside something it touched recently, then most accesses are served from a fast cache and the hundred-cycle trip to DRAM becomes the rare case rather than the common one. Inference performance is decided by how often that bet pays off.

Advertisement

The five levels, with real latency and bandwidth numbers

Here is a representative desktop or laptop x86 core (figures vary by microarchitecture, but the orders of magnitude are what matter). Assume a 3 GHz clock, so 1 cycle ≈ 0.33 ns.

Level	Latency (cycles)	Latency (ns)	Typical size	Approx. bandwidth
Registers	< 1	~0.3	~2 KB (32 × 64 B)	~10 TB/s
L1 data	4–5	~1.3	32–48 KB / core	~1–3 TB/s
L2	12–15	~4–5	256 KB–2 MB / core	~400–900 GB/s
L3 (shared)	40–50	~15–20	8–64 MB	~200–400 GB/s
DRAM	200–350	~80–100	8–128 GB	~50–100 GB/s

Read the table as a cliff, not a slope. Each step down costs roughly 3–4× more latency and gives up a large chunk of bandwidth, and the final step to DRAM is the steepest: an L1 hit is ~1 ns, a DRAM miss ~80 ns, a factor of sixty or more. The size column explains the tension — the fast levels are tiny. A 3.8B-parameter model at 4-bit weighs ~2 GB, which is hundreds of times larger than the entire L3. The model simply cannot live in cache; the only question is how efficiently it streams through it.

Advertisement

Cache lines: memory moves in 64-byte chunks

The hardware never transfers a single byte or a single float from DRAM. It moves a whole cache line — almost universally 64 bytes on x86 and ARM — as the atomic unit. Ask for one 4-byte float and the cache controller fetches the aligned 64-byte block containing it, filling the line and handing you your four bytes.

This single design choice drives most cache behaviour. A 64-byte line holds 16 FP32 values, 32 FP16 values, or 128 INT4 weights, so a program that walks memory sequentially gets those neighbours essentially for free: one miss (the ~80 ns trip) amortizes across the whole line. A program that hops around with a large stride pays the full miss for every access and uses only 4 of every 64 bytes it dragged across the bus — a 16× waste of the scarcest resource in the machine. It also creates false sharing: two threads writing different variables that happen to land on the same line will bounce that line between their private caches as if they were fighting over one value. For inference, the practical rule is that weight matrices should be laid out and traversed so the innermost loop marches contiguously along cache lines, never against them.

Temporal and spatial locality: the two bets caches make

Caches work because real programs are not random. They exhibit two kinds of locality, and every cache mechanism targets one of them. Temporal locality is reuse in time: if you touched an address, you are likely to touch it again soon. Least-recently-used replacement is the cache betting on temporal locality — it evicts what you have not touched in a while on the theory you will not need it. Spatial locality is reuse in space: if you touched an address, you are likely to touch its neighbours soon. The 64-byte line and hardware prefetchers are the cache betting on spatial locality.

Transformer inference has a lopsided locality profile that explains everything downstream. The activations — the vectors flowing between layers — are small and hot, reused across the operations of a layer, so they enjoy strong temporal locality and mostly live in L1/L2. The weights are the opposite: each weight is read, multiplied once, and not touched again until the next token. They have excellent spatial locality (they stream contiguously) but almost no temporal locality at batch size one. A cache cannot help you reuse data you never reuse, which is the seed of the entire bandwidth-bound story.

Why single-stream decode is memory-bandwidth-bound

Autoregressive generation runs one token at a time. To produce a single token, the model must read every weight exactly once and do a small amount of arithmetic with each. There is no reuse to exploit: a weight is loaded, used in one multiply-add, and discarded. So the time to generate a token is bounded below not by how fast the core computes, but by how fast the weights can be streamed out of DRAM.

Concretely, the lower bound is simply time_per_token ≥ weight_bytes / memory_bandwidth. No cache changes this, because the working set (the whole model) is hundreds of times larger than the largest cache — every byte genuinely comes from DRAM. This is the defining fact of CPU SLM inference at batch = 1: it is a memory-bandwidth-bound workload, not a compute-bound one. It is the mirror image of prefill, where the whole prompt is processed at once and the same weights are reused across many token positions, turning the matrix-vector products into matrix-matrix products with real reuse and making prefill compute-bound. Understanding which regime you are in tells you which knob — bandwidth or FLOP/s — will actually move the needle.

A worked bound: milliseconds per token from first principles

Put numbers on it. Take Phi-3-mini (3.8B parameters) quantized to 4-bit, so the weights occupy roughly 2 GB, on a laptop with ~70 GB/s of usable memory bandwidth.

weight_bytes      = 2.0 GB       (3.8B params × 0.5 byte/param at INT4)
memory_bandwidth  = 70 GB/s

t_token (lower bound) = 2.0 GB / 70 GB/s
                      = 0.0286 s
                      ≈ 28.6 ms per token

throughput ceiling    = 1 / 0.0286 s ≈ 35 tokens/second

That ~28 ms is a floor, not a prediction. Real measured rates for this model land around 30–50 ms per token, because the KV cache adds bytes to read, the bandwidth is never 100% utilized, and there is per-token overhead. But the floor is the point: no cleverness in the kernel, no smarter instruction scheduling, no additional cores beyond what saturates the bus can push you below weight_bytes / bandwidth at batch one. When you see a decode speed roughly matching this ratio, the CPU is doing its job and the bottleneck is physics; the only ways down are fewer bytes (quantization) or a wider bus (faster RAM).

Arithmetic intensity: FLOPs per byte

To reason about bandwidth versus compute without guessing, use one number: arithmetic intensity (AI), the ratio of useful floating-point operations to bytes moved from memory.

arithmetic_intensity = FLOPs performed / bytes read from DRAM   (units: FLOP/byte)

AI is a property of an algorithm on a data layout, not of the hardware. It answers: for every byte I am forced to drag across the memory bus, how much math do I get to do? A low AI means you spend most of your time waiting on memory (bandwidth-bound); a high AI means you keep the arithmetic units busy (compute-bound). The decode story above is exactly a statement about AI. A matrix-vector product y = W x with W of shape [n, k] does 2·n·k FLOPs (a multiply and an add per weight) while reading n·k weights of b bytes each. So its intensity is 2·n·k / (b·n·k) = 2/b FLOP/byte — a tiny constant that does not even grow with the matrix size. For FP16 (b = 2) that is 1 FLOP/byte; for INT4 (b = 0.5) it is 4 FLOP/byte. Either way, very low.

The roofline model: one picture that predicts the bottleneck

The roofline model turns arithmetic intensity into a prediction. Plot attainable performance (FLOP/s) against arithmetic intensity (FLOP/byte). Two ceilings bound what any kernel can reach: a slanted roof set by bandwidth, and a flat roof set by peak compute.

attainable_FLOPs_per_s = min( peak_compute,
                             memory_bandwidth × arithmetic_intensity )

Below a certain intensity you are on the slanted part — performance rises linearly with AI because you are bandwidth-limited. Above it you are on the flat part — adding intensity buys nothing because you have hit the compute ceiling. The crossover is the ridge point, and it equals peak_compute / memory_bandwidth, sometimes called the machine's balance. For a CPU that can sustain, say, 300 GFLOP/s over 70 GB/s, the ridge point is 300 / 70 ≈ 4.3 FLOP/byte. Any kernel whose intensity sits left of ~4.3 is memory-bound on this machine; anything to the right is compute-bound. This is why the model is so useful: you compute an algorithm's AI, compare it to one hardware number, and you know your bottleneck before writing the kernel.

Worked example: decode versus prefill on the roofline

Place the two inference phases on that roofline with the ridge point at ~4.3 FLOP/byte.

DECODE  (batch = 1):   y = W x        GEMV, one token
  FLOPs = 2·n·k ,  bytes = b·n·k
  AI = 2/b  = 1 FLOP/byte (FP16)  or  4 FLOP/byte (INT4)
  1 and 4  <  4.3  ridge  ==>  MEMORY-BOUND

PREFILL (N tokens):    Y = X W       GEMM, N rows at once
  FLOPs = 2·N·n·k ,  bytes = b·n·k  (weights read once, reused across N rows)
  AI = 2·N/b
  N = 512, FP16:  AI = 512 FLOP/byte  >>  4.3  ridge  ==>  COMPUTE-BOUND

The only thing that changed is N, the number of token positions sharing the same weight read. Decode processes one position, so each weight earns just 2/b FLOPs before it is thrown away — deep in bandwidth territory. Prefill (or batched serving) reuses each weight across many rows, multiplying the intensity by N and vaulting over the ridge into compute territory. This is the precise, quantitative reason the same model on the same CPU feels compute-bound while ingesting a prompt and memory-bound while typing a reply — and why batching requests is the standard cure for decode inefficiency: it raises N, and therefore AI, at the cost of latency per request.

Cache blocking: turning a bandwidth problem into a cache problem

Prefill and training lean on large matrix-matrix products, and a naive triple loop for C = A · B squanders them. Compute each output element by streaming a full row of A and a full column of B, and for matrices larger than the cache you re-read B from DRAM once per row of A — O(N) passes over the same data. The arithmetic is fine; the memory traffic is catastrophic.

Cache blocking (also called tiling) fixes this by restructuring the loops so the work is done on small sub-blocks that fit in cache. Instead of multiplying whole matrices, you multiply a b × b tile of A by a b × b tile of B into a tile of C, keeping all three tiles resident in a fast cache level and reusing every loaded byte across the whole tile before evicting it. A tile multiply performs 2·b^3 FLOPs while touching only 3·b^2 elements, so its arithmetic intensity is ≈ b — the block size directly sets the intensity. Choose b large enough that the tiles saturate the compute roof, small enough that they fit in cache, and you have converted a bandwidth-bound sprawl into a compute-bound kernel.

Sizing the tiles: fitting three blocks in cache

The block size is not a guess; it falls out of the cache capacity. You need three tiles resident simultaneously — the A tile, the B tile, and the C accumulator tile — so the constraint is 3 · b^2 · bytes_per_element ≤ cache_size. Solve for a 1 MB L2 with FP32 data:

3 · b^2 · 4 bytes  ≤  1,048,576 bytes
        b^2            ≤  87,381
         b             ≤  √87,381  ≈  295

So a block of about 256 × 256 comfortably fits an L2 tile with headroom for the accumulator and other live data. In practice, real GEMM kernels (OpenBLAS, MKL, the kernels inside llama.cpp) use a hierarchy of block sizes — small register tiles that live in the vector registers, mid-size tiles for L1, larger panels for L2 and L3 — nesting the loops so each cache level is reused before the next one down is touched. They also leave margin below the raw capacity because the cache is set-associative: fill it to the brim and conflict misses evict live tile data early. The takeaway is that good matmul performance is a cache-fitting problem first and an arithmetic problem second.