Why CPU SLMs are having a moment
A small language model (SLM) is, loosely, a transformer small enough that its weights fit in ordinary system RAM and its per-token compute fits in a CPU’s budget — today that means roughly 1B to 14B parameters. The case for running them on a CPU is not that the CPU is fast; a mid-range laptop delivers a small fraction of a data-center GPU’s throughput. The case is that single-user, single-stream inference is a fundamentally different workload than the batched serving GPUs were built to exploit.
When you generate tokens one at a time for one user, there is no batch to amortize weight loads across, so the GPU’s enormous arithmetic throughput sits idle waiting on memory. That levels the playing field: the bottleneck becomes moving weights from RAM to the compute units, and a modern CPU with fast DDR5 is not embarrassingly far behind a GPU on that axis. Add that the model now fits in 4–8 GB after quantization, that the CPU is already paid for, and that the data never leaves the device, and the question flips from ‘why would you run this on a CPU?’ to ‘for a single user, why wouldn’t you?’
The memory-bandwidth wall, stated plainly
Everything about CPU SLM performance flows from one distinction between compute-bound and memory-bound work. The roofline model captures it: a kernel’s achievable throughput is min(peak_FLOPs, arithmetic_intensity × memory_bandwidth), where arithmetic intensity is FLOPs / bytes_moved. If a computation does many FLOPs per byte it fetches, it is compute-bound; if few, it is memory-bound and no amount of extra compute helps.
Autoregressive decode at batch size 1 is the memory-bound extreme. To generate one token, a dense model reads every one of its weights exactly once and does about two FLOPs per weight (a multiply and an add in the matrix-vector products). That is an arithmetic intensity of roughly 2 FLOP/byte at FP16 — or 4 FLOP/byte at INT8. A CPU’s ridge point (the intensity at which it stops being memory-bound) is in the tens to low hundreds of FLOP/byte. Decode sits far to the left of that ridge, so the machine spends essentially all its time waiting for weights to arrive from RAM. This single fact — that decode is memory-bound — is the wall against which every other trend in this article is measured.
The decode equation: tokens per second from first principles
Because decode is memory-bound, we can predict its speed with a formula that ignores compute almost entirely. Per generated token you must stream the model’s weights once, so:
bytes_per_token ≈ N_params × bytes_per_param
tokens_per_second ≈ memory_bandwidth / bytes_per_token
= memory_bandwidth / (N_params × bytes_per_param)Work a 7B-parameter model on a laptop with realistic sustained bandwidth of about 60 GB/s (dual-channel DDR5-5600 rarely delivers its ~90 GB/s theoretical peak). At FP16, bytes_per_param = 2, so the model is 14 GB and you get 60/14 ≈ 4.3 tokens/s — sluggish. At INT8 (7 GB): ~8.6 tokens/s. At 4-bit (~3.7 GB): ~16 tokens/s — faster than most people read. The formula is why quantization is the master lever for CPU inference: halving bytes-per-weight roughly doubles decode speed, directly, with no new hardware. It also sets a ceiling honesty demands we respect — you cannot beat bandwidth / model_size no matter how clever the kernel.
Quantization I: the bits-per-weight lever and sub-4-bit
Quantization stores each weight in fewer bits than its trained FP16/BF16 representation. The workhorse is affine integer quantization: within a small block of weights you find the range, pick a scale s and zero-point z, and store q = round(w/s) + z as a low-bit integer, reconstructing w ≈ s · (q - z). Blockwise scales (one s per 32 or 64 weights) keep a few outliers from wrecking the whole tensor.
The frontier is sub-4-bit. Methods like GPTQ and AWQ push to 3–4 bits by accounting for which weights matter most to the layer’s output — AWQ scales up salient channels before rounding; GPTQ minimizes output error greedily using second-order (Hessian) information. QuIP# and similar incoherence-processing methods reach 2 bits with surprisingly small quality loss. The most radical is BitNet b1.58, which trains weights to be ternary {-1, 0, +1} — log2(3) ≈ 1.58 bits each — turning the dominant matrix multiply into pure additions and subtractions, no floating-point multiplier needed. That is almost custom-made for CPUs, whose integer add throughput dwarfs their float-multiply throughput.
Quantization II: QAT versus post-training, and why QAT wins low
There are two ways to get a quantized model. Post-training quantization (PTQ) takes a finished FP16 model and rounds it, optionally calibrating scales on a few hundred sample inputs. It is cheap and needs no training data, and down to about 4 bits it costs very little accuracy. Below 4 bits, though, the rounding error grows faster than clever scale-picking can absorb, and PTQ models start to visibly degrade.
Quantization-aware training (QAT) fixes this by simulating the quantization during training or fine-tuning: the forward pass rounds weights to low bits, but gradients flow through a straight-through estimator that treats the non-differentiable rounding as the identity, so the network learns weights that are robust to being rounded. The model effectively pre-compensates for the error it will suffer at inference. QAT is what makes 2-bit and ternary models usable rather than merely small — BitNet, for instance, is trained ternary from scratch, not rounded after the fact. The trend line is stark: the number of bits per weight needed for near-lossless quality has fallen from 16 in 2021 to about 4 by 2023 to roughly 2 by 2025 — call it a halving every 18-or-so months, a Moore’s-law cadence for model density that directly doubles CPU decode speed each step.
Stronger small models: over-training past Chinchilla
Quantization shrinks a fixed model; the second force makes the small model itself smarter. The Chinchilla scaling laws found that, for a fixed training budget, compute-optimal training uses about 20 tokens per parameter. But that optimum minimizes training cost, not inference cost — and for a model you will serve billions of times on-device, inference cost is what matters.
So the industry deliberately over-trains small models: pour vastly more data into a smaller network than compute-optimality suggests. Llama 3 8B was trained on about 15 trillion tokens — roughly 1,875 tokens per parameter, nearly 100× the Chinchilla ratio. The result is an 8B model that would have been mistaken for a much larger one a couple of years earlier. The economic logic is clean: you pay the extra training compute once, and every user forever gets a smaller model that is cheaper and faster to run. For CPU SLMs this is decisive — it means the 3B–8B class keeps absorbing capability that used to require 30B+, moving genuinely useful models into the size range that fits in laptop RAM at interactive speed.
Distillation: compressing a big teacher into a small student
Distillation is the other route to a strong small model. Instead of training the student only on hard one-hot labels, you train it to match a large teacher’s full output distribution — its soft targets. The loss is typically a KL divergence between the teacher’s and student’s next-token distributions, often with a temperature T that softens both so the student learns the teacher’s relative preferences among all tokens, not just the top one:
L = KL( softmax(z_teacher / T) || softmax(z_student / T) )Those soft targets carry far more information per example than a single correct token — they encode that ‘Paris’ is likely, ‘Lyon’ plausible, and ‘banana’ absurd — so the student converges to good behavior with less data and fewer parameters than training from scratch would need. Modern recipes lean on this heavily: use a frontier model to generate high-quality synthetic training data and to supply soft labels, then distill into a compact student. Much of why today’s 1B–3B models feel disproportionately capable is that they were taught by something far larger — a direct pipeline from the data center into the model on your CPU.
Architectural efficiency I: GQA and the KV-cache shrink
Decode speed depends on how many bytes you stream per token, and it is not only weights — the KV cache also grows with context and must be read every step. Vanilla multi-head attention stores a separate key and value vector per head per token, so the cache is 2 × L × n_heads × d_head bytes per token × sequence length — at long context this rivals the weights.
Grouped-query attention (GQA) shrinks it. Instead of one key/value pair per query head, several query heads share one key/value head. With 32 query heads mapped onto 8 KV groups, the KV cache shrinks 32/8 = 4× with almost no quality loss; multi-query attention (MQA) is the extreme, a single KV head. Because attention quality depends much more on the query heads than on having independent keys and values, this is nearly free accuracy-wise but directly cuts the bytes-per-token you stream during long-context decode. For a CPU SLM doing retrieval-augmented generation or long chats, GQA is what keeps the memory-bound decode from getting slower as the conversation grows — it is now standard in essentially every model built to be served efficiently.
Architectural efficiency II: MoE with small active params
Mixture-of-experts (MoE) decouples a model’s total size from its per-token cost. Each transformer block’s feed-forward layer is replaced by many parallel ‘expert’ FFNs plus a small router that, per token, selects only the top-k experts (often 2). The token flows through only those, so the active parameter count is a fraction of the total. Mixtral 8×7B, for example, holds about 47B parameters but activates only ~13B per token.
The catch, on a CPU, is subtle and worth stating honestly. MoE cuts FLOPs cleanly, but decode is memory-bound, not compute-bound — and the bytes you stream per token are the routed experts’ weights, which change token to token. You still need all experts resident in RAM (so the memory footprint is the full 47B), and expert selection has poor locality, defeating caches. The win is real — bytes-per-token tracks active, not total, params — but MoE trades a larger RAM footprint for that speed, which is exactly the resource a laptop is shortest on. MoE is a superb fit when you have the RAM to spare and want frontier quality at small-model decode cost; it is an awkward one on a tightly memory-limited device.
Architectural efficiency III: linear and state-space models
Attention’s KV cache grows with sequence length — O(N) memory read that grows every token. State-space models (SSMs) like Mamba attack this at the root. An SSM carries a fixed-size recurrent state h and updates it per token: h_t = Ā h_{t-1} + B̄ x_t, y_t = C h_t. Crucially the state size is constant — it does not grow with context — so generation is O(1) memory and O(N) total time, versus attention’s growing cache and O(N^2) prefill.
For a CPU that means the bytes-per-token during decode stay flat no matter how long the conversation, which is precisely the property a memory-bound machine wants. The honest caveat is that pure SSMs can be weaker at exact recall — pulling a specific token from far back — because they compress history into a fixed state rather than keeping every key. The pragmatic answer, now common, is hybrid models that interleave a few full-attention layers among many SSM (or linear-attention) layers, keeping most of the constant-memory benefit while retaining sharp recall where it counts. For long-context on-device inference, this hybrid direction is one of the most promising things on the horizon.
CPU hardware I: AVX-512, VNNI, and AMX
CPUs did not stand still. The vector units widened and grew instructions aimed squarely at the low-precision integer math that quantized inference needs. AVX-512 processes 512-bit vectors — 64 INT8 values at once per lane. VNNI (Vector Neural Network Instructions) adds a fused multiply-accumulate over INT8 operands that does in one instruction what previously took several, roughly tripling INT8 dot-product throughput.
The bigger jump is Intel’s AMX (Advanced Matrix Extensions): a set of 2D tile registers with a dedicated matrix-multiply unit (TMUL) that computes a whole tile-by-tile INT8 or BF16 matmul per instruction — on the order of 1024 multiply-accumulates in a single op per core per cycle. That is a large multiplier on the CPU’s effective throughput for exactly the operation transformers are made of. The important framing, though: these units raise the CPU’s compute ceiling and its ridge point, which matters enormously for compute-bound prefill (digesting a long prompt in parallel). They do far less for memory-bound single-token decode, which remains gated by bandwidth. Great hardware, but pointed at the half of the problem that was never the bottleneck for generation.
CPU hardware II: caches and the bandwidth that actually gates you
Since decode is bandwidth-bound, the memory subsystem is the hardware that matters most, and it improved on two fronts. First, bandwidth: DDR5 roughly doubled per-channel throughput over DDR4, and dual-channel DDR5-5600 offers about 90 GB/s theoretical (call it 55–70 sustained). Apple’s M-series and other unified-memory designs go much further — the wide LPDDR5 buses on high-end parts reach several hundred GB/s, which is exactly why those machines punch so far above typical laptops at local LLM decode. Bandwidth maps almost linearly to tokens/s, so this is the number to watch.
Second, caches: last-level caches grew, and designs with a large stacked cache can keep a meaningful slice of a small quantized model on-die. When weights hit in cache instead of DRAM, effective bandwidth jumps several-fold and the roofline lifts. This is a real, if partial, escape from the DRAM wall — it helps the hottest, most-reused weights and the KV cache, though a multi-gigabyte model still overflows any cache and must stream from DRAM. The trajectory is encouraging: every generation adds bandwidth and cache, and unlike raw clock speed, both translate directly into on-device tokens/s.
CPU hardware III: on-device NPUs and heterogeneous inference
Recent laptop and phone SoCs ship a third compute block beside the CPU and GPU: a neural processing unit (NPU), a low-power accelerator built for INT8/INT4 matrix math, now commonly rated at 40+ TOPS. Its point is not raw peak — a discrete GPU still wins there — but performance per watt, which is what governs a fanless device running a model without draining the battery.
The realistic near-future is heterogeneous: split the workload across blocks by what each does best. The NPU or AMX-equipped CPU handles the compute-heavy, parallel prefill; the CPU with its fast path to full system RAM handles memory-bound decode, since the NPU’s own tight memory often cannot hold the whole model. Because prefill and decode have opposite bottlenecks — compute-bound versus bandwidth-bound — matching each phase to the block that suits it is a natural, and increasingly automated, division of labor. The strategic effect is that ‘the CPU’ is becoming a coordinated cluster of specialized units on one chip, and on-device inference the workload that finally exercises all of them together.