Running a language model on a CPU is not a fallback for people without a GPU — it is a design space with its own rules, and once you know the rules a modern laptop or server socket serves a 7–8B model at genuinely useful speeds. The whole story is a single sentence you can carry through this article: prefill is compute-bound, decode is memory-bandwidth-bound. Every optimization that matters on a CPU — quantizing weights, pinning threads to NUMA nodes, batching requests, choosing intra-op vs inter-op parallelism — is either feeding more work to the vector units during prefill or moving fewer bytes across the memory bus during decode. This piece walks the end-to-end path a request takes: tokenize → prefill → decode loop → detokenize. It explains why the decode loop, not the math, is the bottleneck; derives a concrete tokens-per-second estimate for a real CPU and an 8B model from first principles; and points at the frameworks — llama.cpp/GGML, ONNX Runtime, OpenVINO — that turn the theory into a working server.
The serving path, end to end
A request to a CPU inference server is not one operation; it is a small pipeline with four distinct stages, and each has a different cost profile. First tokenization turns the input string into a list of integer token IDs. Then prefill runs one forward pass over the entire prompt at once, building the key/value cache and producing the logits for the first new token. Then the decode loop runs one forward pass per output token, each pass consuming the single previous token and the growing KV cache. Finally detokenization maps the generated IDs back to text.
The shape of the workload is lopsided. Tokenize and detokenize are cheap string operations. Prefill is one big, highly parallel matrix-multiply pass whose cost is roughly proportional to prompt_length. The decode loop is where the wall-clock time for a chat response actually goes: generating 400 tokens means 400 sequential forward passes, each of which must stream a large fraction of the model’s weights out of RAM. Understanding a CPU pipeline means understanding why those two middle stages behave so differently — and optimizing one rarely helps the other.
Tokenization: cheap, but not free
Tokenization converts text to IDs using a learned subword vocabulary — typically byte-pair encoding (BPE) or a Unigram model, as shipped in SentencePiece or the tokenizers library. A greedy or priority-queue merge process repeatedly combines the most frequent adjacent pair until no learned merge applies, mapping, say, "tokenization" to a handful of subword pieces. The output is a vector of IDs, shape [N] for a prompt of N tokens.
On a CPU this stage is essentially free relative to the model forward pass — microseconds to low milliseconds — but it deserves two cautions. First, the tokenizer must match the model exactly; a mismatched vocabulary or a missing special token (BOS/EOS, chat-template markers) silently corrupts the input and the model produces nonsense. Second, token count is not word count: English averages roughly 1 token ≈ 0.75 words, code and non-Latin scripts inflate far higher. Because every downstream cost scales with N, an accurate token count is the unit in which you budget latency and KV memory. Detokenization at the end is the inverse map and is equally cheap, with the one subtlety that multi-byte UTF-8 characters may span token boundaries and must be buffered until complete before streaming to the user.
Prefill: one pass, compute-bound
Prefill processes the whole prompt in a single forward pass. Because all N prompt positions are available at once, the per-layer work is dense matrix multiplication: the activation matrix X: [N, d] is multiplied by each weight matrix, and every weight is reused across all N rows. That reuse is the whole point. A weight matrix of [d, d] is read from memory once but participates in N×d×d multiply-adds, so the ratio of arithmetic to bytes moved — the arithmetic intensity — is high, scaling with N.
The FLOP count is easy to estimate. A dense transformer does about 2 × P floating-point operations per token in the matmul-dominated path, where P is the parameter count (the factor of 2 is one multiply plus one add per weight). For a prompt of N tokens that is 2 × P × N FLOPs, plus the O(N^2) attention term that matters only for long prompts. Because the vector units are the bottleneck and memory traffic is amortized over N tokens, prefill is compute-bound: it is limited by how many fused multiply-adds per second the cores can retire. This is why prefill throughput (tokens/second) on a CPU can be an order of magnitude higher than decode throughput, and why time-to-first-token grows with prompt length while inter-token latency does not.
The decode loop: one token at a time
Decode is autoregressive and inherently sequential. To produce token t+1 the model needs token t, which did not exist until the previous step finished. So generation is a loop: run a forward pass on a single new position, sample the next token from the output logits, append it to the sequence, repeat until an EOS token or a length limit. Each iteration’s activation is a matrix of shape [1, d] — one row, not N.
That single-row shape is the crux of CPU inference. When you multiply a [1, d] activation by a [d, d] weight matrix, each weight is used exactly once before being discarded. The arithmetic intensity collapses to roughly one multiply-add per weight loaded — on the order of 0.5–2 FLOPs per byte. The cores are starved: they finish the tiny amount of math long before the next slab of weights arrives from RAM. The forward pass is no longer limited by compute; it is limited by how fast the memory subsystem can deliver the model’s weights. Every decode step reads (nearly) the entire model from memory, once. That single fact drives almost every CPU-inference optimization worth knowing.
Why decode is memory-bandwidth-bound
The clean way to see this is the roofline model. A machine has two ceilings: peak compute (FLOP/s) and peak memory bandwidth (bytes/s). Their ratio is the machine balance or ridge point, in FLOPs per byte. A kernel whose arithmetic intensity sits below the ridge point is memory-bound — it cannot reach peak FLOP/s because it runs out of bytes; above it, compute-bound.
ridge_point = peak_FLOPs / peak_bandwidth
e.g. 1.8 TFLOP/s / 75 GB/s ≈ 24 FLOP/byte
decode intensity (batch 1) ≈ 0.5 - 2 FLOP/byte → << 24 → memory-bound
prefill intensity (N tokens) scales with N → > 24 → compute-boundDecode’s intensity of ~1 is far below a typical CPU ridge point of 20–40, so the vector units idle most of the cycle waiting on DRAM. The consequence is liberating once you accept it: during decode, adding more cores or faster clocks barely helps, because the cores were never the bottleneck. What helps is moving fewer bytes per token (quantize the weights) or moving bytes faster (more memory channels, faster DIMMs, staying on one NUMA node). This is also why decode throughput on a laptop tracks its memory bandwidth far more tightly than its core count — a number people find surprising until they have internalized the roofline.
A back-of-the-envelope: bandwidth sets the ceiling
Here is the estimate every CPU-inference decision leans on. If decode reads the whole model once per token, then peak decode throughput is simply bandwidth divided by model size in bytes.
tokens_per_sec ≈ achievable_bandwidth / bytes_read_per_token
≈ achievable_bandwidth / model_size_bytesThe formula ignores the KV cache read (small for short context) and any compute time (negligible in the memory-bound regime), which is exactly why it is a ceiling: real throughput is 60–90% of it. But it predicts the leading digit correctly, and it explains the single biggest lever — model size in bytes — which is precisely what quantization attacks. Halve the bytes per weight and you roughly double decode throughput, with no change to the math being performed. The same formula run backwards is a sizing tool: given a target of, say, 20 tokens/second and a 75 GB/s machine, your model must fit in about 75 / 20 = 3.75 GB, which tells you immediately that an 8B model must be quantized to roughly 4 bits to hit the target. We work the full numbers below.
Worked example: an 8B model on a desktop CPU
Take a concrete machine: a modern desktop with an AVX-512 CPU and dual-channel DDR5-6000. Theoretical bandwidth is 6000 MT/s × 8 bytes × 2 channels = 96 GB/s; call the achievable figure ~75 GB/s after real-world efficiency. The model is Llama-3.1-8B quantized to Q4_K_M — about 4.5 effective bits per weight, so 8e9 × 4.5 / 8 ≈ 4.5 GB of weights.
DECODE (memory-bound):
peak_tok/s = 75 GB/s / 4.5 GB ≈ 16.7 tok/s (ceiling)
realistic = ~11 - 15 tok/s (60-90% of ceiling)
PREFILL (compute-bound), assume ~1.8 TFLOP/s sustained:
flops/token = 2 × 8e9 = 1.6e10 FLOP
prefill_tok/s = 1.8e12 / 1.6e10 ≈ 110 tok/s
TTFT for a 512-token prompt ≈ 512 / 110 ≈ 4.6 sRead the contrast. Prefill chews through the prompt at ~110 tok/s; decode crawls at ~13 tok/s — nearly an order of magnitude apart, exactly as the roofline predicts. A user asking a short question and getting a 300-token answer waits a fraction of a second for prefill and then ~20–25 seconds streaming the reply, at a readable pace faster than most people read. These are order-of-magnitude figures — kernels, cache effects, and thermals move them — but the structure is robust: on a CPU, decode speed is your DDR bandwidth divided by your quantized model size, full stop.
Weight quantization: fewer bytes per token
Because decode throughput is bandwidth over model size, shrinking the model in bytes is the highest-leverage CPU optimization there is. Quantization stores weights in fewer bits than FP16’s sixteen: INT8 halves the bytes, 4-bit schemes quarter them. The llama.cpp/GGUF family — Q4_K_M, Q5_K_M, Q8_0 and friends — stores weights in small blocks (say 32 values) with a shared scale (and sometimes a min), so a block of 4-bit codes plus a 16-bit scale averages ~4.5 bits per weight.
The mechanism that matters: weights are stored quantized and dequantized on the fly into vector registers just before the matmul, or fed to integer-mixed kernels. Either way the bytes crossing the memory bus are the quantized ones, which is the quantity the roofline cares about. Going from FP16 (~16 GB for 8B) to Q4 (~4.5 GB) cuts bytes-per-token by ~3.5× and lifts decode throughput by nearly the same factor. The cost is accuracy: 8-bit is essentially lossless, 4-bit is a small and usually acceptable quality drop for chat, and 2–3 bit schemes degrade noticeably. The k-quants and importance-weighted variants (imatrix) spend their limited bit budget on the weights that matter most, which is why a good 4-bit quant is far better than naive rounding suggests.
Thread pools: intra-op vs inter-op parallelism
CPU runtimes expose two knobs for parallelism, and confusing them is a classic tuning mistake. Intra-op parallelism splits a single operator — one big matrix multiply — across many threads, each computing a slice of the output. Inter-op parallelism runs independent operators in the graph concurrently on different threads. In ONNX Runtime these are intra_op_num_threads and inter_op_num_threads; llama.cpp exposes an intra-op-style thread count via -t/n_threads.
For transformer decode, intra-op is what pays off: each step is a chain of large matmuls with little independent work to overlap, so you want all threads cooperating on each matmul in turn. Two counter-intuitive rules follow from the memory-bound nature of decode. First, more threads is not always faster — once threads saturate the memory bus, extra threads only add contention and synchronization overhead, so the optimum is often the number of physical cores on one socket, not the hyperthread count. Second, prefill (compute-bound) scales with cores far better than decode (bandwidth-bound) does, so the ‘best’ thread count can differ between the two phases. Oversubscribing threads or fighting another thread pool (BLAS spawning its own) is a common cause of mysteriously slow inference.
NUMA: locality is bandwidth
On multi-socket servers — and even some large single-socket parts — memory is non-uniform (NUMA): each socket has its own memory controllers and local DIMMs, and reaching another socket’s memory crosses an interconnect that is slower and lower-bandwidth than local access. Since decode is bandwidth-bound, NUMA effects hit it directly: if a thread on socket 0 reads weights that live in socket 1’s DRAM, it pays a remote-access penalty on every token.
The fixes are about locality. Pin the inference threads to the cores of one NUMA node and allocate the weights on that same node so every access is local — on Linux this is numactl --cpunodebind=0 --membind=0. For a model too large for one node’s memory, the alternatives are to replicate weights per node (trading RAM for bandwidth) or to accept a first-touch allocation policy that at least keeps each thread’s working set local. The general principle: a two-socket box does not give you double the decode throughput unless you keep each socket working on local memory. Treated naively, the interconnect becomes the bottleneck and the second socket adds latency instead of speed. NUMA awareness is one of the largest — and most overlooked — server-CPU levers.
SIMD and the GEMM underneath
Underneath every framework, the matmul is done by vectorized kernels. Modern x86 CPUs provide SIMD instruction sets — AVX2 (256-bit), AVX-512 (512-bit), and the AVX-512-VNNI / AMX extensions for fast integer dot products — that let one instruction perform many multiply-adds at once. ARM offers NEON and SVE. A fused multiply-add on a 512-bit register handles 16 FP32 or 32 BF16 lanes per instruction, and VNNI/AMX push INT8 throughput far higher, which is exactly why quantized models can also be faster in the compute-bound prefill phase, not just smaller.
Getting near peak requires a well-tuned GEMM (general matrix multiply) that blocks the computation for the L1/L2/L3 cache hierarchy and keeps the vector units fed — the domain of libraries like Intel oneDNN (and its oneMKL BLAS), OpenBLAS, and the hand-written kernels inside GGML. This is also where instruction-set targeting matters: a binary compiled only for AVX2 leaves half the width of an AVX-512 machine unused, and a model runner that dispatches to the widest available kernel at runtime can be markedly faster than a generic build. The takeaway for prefill: throughput is set by how efficiently the GEMM saturates the widest SIMD path your CPU has.
KV cache management on the CPU
The KV cache is the memory of the decode loop: rather than recomputing attention over the whole sequence each step, the keys and values of every past token are cached, so a new token attends to them directly. Its size grows linearly with context: roughly 2 × n_layers × n_kv_heads × head_dim × seq_len × bytes_per_elem. For an 8B model at a few thousand tokens this is on the order of a gigabyte in FP16 — a real slice of RAM and, at long context, a real slice of the bytes read per decode step.
Several CPU-relevant moves follow. Grouped-query attention (fewer KV heads than query heads) shrinks the cache directly and is now standard in efficient models. KV-cache quantization (storing K and V in 8-bit or 4-bit) trades a little accuracy for less memory and less bandwidth per token, which matters more the longer the context. And because prompt prefill is expensive, prompt/prefix caching — reusing the KV cache of a shared system prompt across requests — can eliminate most of the per-request prefill cost in a chat server. Managing the KV cache well is what keeps long-context CPU inference from silently sliding from ‘model-size-bound’ to ‘cache-bound’ as conversations grow.
Batching: buying back arithmetic intensity
Batching is the one move that changes decode’s fundamental character. Recall that batch-1 decode reads each weight and uses it once — intensity ~1. If you process B independent sequences together, the activation becomes [B, d] and each weight, read once, now serves B multiply-adds. Arithmetic intensity rises roughly linearly with B, sliding the workload up the roofline from memory-bound toward the compute-bound ridge point.
The payoff is throughput: at batch 1 you get ~13 tok/s for one user; at batch 8, aggregate throughput across the eight streams can approach 8× that, because you amortize each weight read over eight tokens without reading more bytes. This is why a CPU serving many concurrent chats wants continuous batching — dynamically grouping whatever requests are decoding right now — even though it does nothing for a single user’s latency. The limits are two: batching raises per-token latency slightly (the batch moves at the speed of its members) and it multiplies KV-cache memory by B, which can become the new ceiling. On CPU the sweet spot is usually a modest batch — enough to lift intensity toward the ridge point, not so much that KV cache exhausts RAM. Beyond the ridge point, extra batch stops helping decode because you have become compute-bound again.
Frameworks: llama.cpp, ONNX Runtime, OpenVINO
Three ecosystems dominate CPU inference, each a different point on the control-versus-convenience axis. llama.cpp / GGML is the purpose-built option: a dependency-light C/C++ engine with hand-tuned SIMD kernels, the GGUF quantized-weight format, first-class 4-bit/5-bit k-quants, KV-cache quantization, and continuous batching in its server. For running a quantized LLM/SLM on a laptop or a commodity server it is often the fastest and simplest path, and its quant formats are the de-facto standard.
ONNX Runtime is the general-purpose inference engine: export any model to ONNX and run it through a graph optimizer with pluggable execution providers, the CPU provider backed by oneDNN/MLAS kernels. It exposes the intra-op / inter-op thread controls directly and shines when you need one runtime for many model types across platforms. OpenVINO is Intel’s inference stack, tuned hard for Intel CPUs (and iGPUs/NPUs), with aggressive graph compilation, INT8 quantization tooling, and strong throughput on Xeon hardware. A reasonable default: reach for llama.cpp to serve a quantized chat model quickly, ONNX Runtime for portable multi-model serving, and OpenVINO when you are squeezing maximum throughput from Intel silicon. All three obey the same physics — they differ in kernels, formats, and ergonomics, not in the roofline.
Putting the pipeline together: a tuning checklist
Assembling a fast CPU server is a sequence of decisions, each justified by the physics above. Quantize the weights — usually 4-bit k-quants — because decode throughput is bandwidth over model size and this is the biggest lever. Set threads to physical cores on one socket, not the hyperthread count, since decode saturates the bus before it saturates the cores. Pin to one NUMA node and allocate weights locally so every byte read is local bandwidth. Use a build that targets your widest SIMD (AVX-512/VNNI/AMX) so prefill and dequant run at full width.
Then match the serving strategy to the workload. A single interactive user wants low latency: batch 1, quantized KV cache for long chats, prefix caching for a shared system prompt. A multi-user service wants throughput: continuous batching to lift arithmetic intensity, sized so aggregate KV cache still fits in RAM. Measure the two phases separately — time-to-first-token is a prefill (compute) number, inter-token latency is a decode (bandwidth) number — because a change that helps one can be neutral or harmful to the other. Do this, and a plain CPU serves a small language model at a speed that surprises people who assumed a GPU was mandatory.