For a decade the default answer to ‘where does the model run?’ was ‘on a GPU, in someone else’s data center.’ That answer is quietly eroding. A stack of independent trends — quantization pushing below four bits per weight, small models trained far past the compute-optimal point, architectures that shrink the state you must stream, and CPUs that grew matrix units, wider vectors, and faster memory — are converging on a world where a capable language model runs on the laptop or phone already in your hand, with no network round trip and no token bill. This piece is a technically grounded tour of those forces. It is deliberately not hype: the single most important number in on-device inference is memory bandwidth, and every optimistic claim here is checked against that wall. We will do the arithmetic, work a concrete tokens-per-second estimate, and be honest about what the CPU can and cannot do.
Why CPU SLMs are having a moment
A small language model (SLM) is, loosely, a transformer small enough that its weights fit in ordinary system RAM and its per-token compute fits in a CPU’s budget — today that means roughly 1B to 14B parameters. The case for running them on a CPU is not that the CPU is fast; a mid-range laptop delivers a small fraction of a data-center GPU’s throughput. The case is that single-user, single-stream inference is a fundamentally different workload than the batched serving GPUs were built to exploit.
When you generate tokens one at a time for one user, there is no batch to amortize weight loads across, so the GPU’s enormous arithmetic throughput sits idle waiting on memory. That levels the playing field: the bottleneck becomes moving weights from RAM to the compute units, and a modern CPU with fast DDR5 is not embarrassingly far behind a GPU on that axis. Add that the model now fits in 4–8 GB after quantization, that the CPU is already paid for, and that the data never leaves the device, and the question flips from ‘why would you run this on a CPU?’ to ‘for a single user, why wouldn’t you?’
The memory-bandwidth wall, stated plainly
Everything about CPU SLM performance flows from one distinction between compute-bound and memory-bound work. The roofline model captures it: a kernel’s achievable throughput is min(peak_FLOPs, arithmetic_intensity × memory_bandwidth), where arithmetic intensity is FLOPs / bytes_moved. If a computation does many FLOPs per byte it fetches, it is compute-bound; if few, it is memory-bound and no amount of extra compute helps.
Autoregressive decode at batch size 1 is the memory-bound extreme. To generate one token, a dense model reads every one of its weights exactly once and does about two FLOPs per weight (a multiply and an add in the matrix-vector products). That is an arithmetic intensity of roughly 2 FLOP/byte at FP16 — or 4 FLOP/byte at INT8. A CPU’s ridge point (the intensity at which it stops being memory-bound) is in the tens to low hundreds of FLOP/byte. Decode sits far to the left of that ridge, so the machine spends essentially all its time waiting for weights to arrive from RAM. This single fact — that decode is memory-bound — is the wall against which every other trend in this article is measured.
The decode equation: tokens per second from first principles
Because decode is memory-bound, we can predict its speed with a formula that ignores compute almost entirely. Per generated token you must stream the model’s weights once, so:
bytes_per_token ≈ N_params × bytes_per_param
tokens_per_second ≈ memory_bandwidth / bytes_per_token
= memory_bandwidth / (N_params × bytes_per_param)Work a 7B-parameter model on a laptop with realistic sustained bandwidth of about 60 GB/s (dual-channel DDR5-5600 rarely delivers its ~90 GB/s theoretical peak). At FP16, bytes_per_param = 2, so the model is 14 GB and you get 60/14 ≈ 4.3 tokens/s — sluggish. At INT8 (7 GB): ~8.6 tokens/s. At 4-bit (~3.7 GB): ~16 tokens/s — faster than most people read. The formula is why quantization is the master lever for CPU inference: halving bytes-per-weight roughly doubles decode speed, directly, with no new hardware. It also sets a ceiling honesty demands we respect — you cannot beat bandwidth / model_size no matter how clever the kernel.
Quantization I: the bits-per-weight lever and sub-4-bit
Quantization stores each weight in fewer bits than its trained FP16/BF16 representation. The workhorse is affine integer quantization: within a small block of weights you find the range, pick a scale s and zero-point z, and store q = round(w/s) + z as a low-bit integer, reconstructing w ≈ s · (q - z). Blockwise scales (one s per 32 or 64 weights) keep a few outliers from wrecking the whole tensor.
The frontier is sub-4-bit. Methods like GPTQ and AWQ push to 3–4 bits by accounting for which weights matter most to the layer’s output — AWQ scales up salient channels before rounding; GPTQ minimizes output error greedily using second-order (Hessian) information. QuIP# and similar incoherence-processing methods reach 2 bits with surprisingly small quality loss. The most radical is BitNet b1.58, which trains weights to be ternary {-1, 0, +1} — log2(3) ≈ 1.58 bits each — turning the dominant matrix multiply into pure additions and subtractions, no floating-point multiplier needed. That is almost custom-made for CPUs, whose integer add throughput dwarfs their float-multiply throughput.
Quantization II: QAT versus post-training, and why QAT wins low
There are two ways to get a quantized model. Post-training quantization (PTQ) takes a finished FP16 model and rounds it, optionally calibrating scales on a few hundred sample inputs. It is cheap and needs no training data, and down to about 4 bits it costs very little accuracy. Below 4 bits, though, the rounding error grows faster than clever scale-picking can absorb, and PTQ models start to visibly degrade.
Quantization-aware training (QAT) fixes this by simulating the quantization during training or fine-tuning: the forward pass rounds weights to low bits, but gradients flow through a straight-through estimator that treats the non-differentiable rounding as the identity, so the network learns weights that are robust to being rounded. The model effectively pre-compensates for the error it will suffer at inference. QAT is what makes 2-bit and ternary models usable rather than merely small — BitNet, for instance, is trained ternary from scratch, not rounded after the fact. The trend line is stark: the number of bits per weight needed for near-lossless quality has fallen from 16 in 2021 to about 4 by 2023 to roughly 2 by 2025 — call it a halving every 18-or-so months, a Moore’s-law cadence for model density that directly doubles CPU decode speed each step.
Stronger small models: over-training past Chinchilla
Quantization shrinks a fixed model; the second force makes the small model itself smarter. The Chinchilla scaling laws found that, for a fixed training budget, compute-optimal training uses about 20 tokens per parameter. But that optimum minimizes training cost, not inference cost — and for a model you will serve billions of times on-device, inference cost is what matters.
So the industry deliberately over-trains small models: pour vastly more data into a smaller network than compute-optimality suggests. Llama 3 8B was trained on about 15 trillion tokens — roughly 1,875 tokens per parameter, nearly 100× the Chinchilla ratio. The result is an 8B model that would have been mistaken for a much larger one a couple of years earlier. The economic logic is clean: you pay the extra training compute once, and every user forever gets a smaller model that is cheaper and faster to run. For CPU SLMs this is decisive — it means the 3B–8B class keeps absorbing capability that used to require 30B+, moving genuinely useful models into the size range that fits in laptop RAM at interactive speed.
Distillation: compressing a big teacher into a small student
Distillation is the other route to a strong small model. Instead of training the student only on hard one-hot labels, you train it to match a large teacher’s full output distribution — its soft targets. The loss is typically a KL divergence between the teacher’s and student’s next-token distributions, often with a temperature T that softens both so the student learns the teacher’s relative preferences among all tokens, not just the top one:
L = KL( softmax(z_teacher / T) || softmax(z_student / T) )Those soft targets carry far more information per example than a single correct token — they encode that ‘Paris’ is likely, ‘Lyon’ plausible, and ‘banana’ absurd — so the student converges to good behavior with less data and fewer parameters than training from scratch would need. Modern recipes lean on this heavily: use a frontier model to generate high-quality synthetic training data and to supply soft labels, then distill into a compact student. Much of why today’s 1B–3B models feel disproportionately capable is that they were taught by something far larger — a direct pipeline from the data center into the model on your CPU.
Architectural efficiency I: GQA and the KV-cache shrink
Decode speed depends on how many bytes you stream per token, and it is not only weights — the KV cache also grows with context and must be read every step. Vanilla multi-head attention stores a separate key and value vector per head per token, so the cache is 2 × L × n_heads × d_head bytes per token × sequence length — at long context this rivals the weights.
Grouped-query attention (GQA) shrinks it. Instead of one key/value pair per query head, several query heads share one key/value head. With 32 query heads mapped onto 8 KV groups, the KV cache shrinks 32/8 = 4× with almost no quality loss; multi-query attention (MQA) is the extreme, a single KV head. Because attention quality depends much more on the query heads than on having independent keys and values, this is nearly free accuracy-wise but directly cuts the bytes-per-token you stream during long-context decode. For a CPU SLM doing retrieval-augmented generation or long chats, GQA is what keeps the memory-bound decode from getting slower as the conversation grows — it is now standard in essentially every model built to be served efficiently.
Architectural efficiency II: MoE with small active params
Mixture-of-experts (MoE) decouples a model’s total size from its per-token cost. Each transformer block’s feed-forward layer is replaced by many parallel ‘expert’ FFNs plus a small router that, per token, selects only the top-k experts (often 2). The token flows through only those, so the active parameter count is a fraction of the total. Mixtral 8×7B, for example, holds about 47B parameters but activates only ~13B per token.
The catch, on a CPU, is subtle and worth stating honestly. MoE cuts FLOPs cleanly, but decode is memory-bound, not compute-bound — and the bytes you stream per token are the routed experts’ weights, which change token to token. You still need all experts resident in RAM (so the memory footprint is the full 47B), and expert selection has poor locality, defeating caches. The win is real — bytes-per-token tracks active, not total, params — but MoE trades a larger RAM footprint for that speed, which is exactly the resource a laptop is shortest on. MoE is a superb fit when you have the RAM to spare and want frontier quality at small-model decode cost; it is an awkward one on a tightly memory-limited device.
Architectural efficiency III: linear and state-space models
Attention’s KV cache grows with sequence length — O(N) memory read that grows every token. State-space models (SSMs) like Mamba attack this at the root. An SSM carries a fixed-size recurrent state h and updates it per token: h_t = Ā h_{t-1} + B̄ x_t, y_t = C h_t. Crucially the state size is constant — it does not grow with context — so generation is O(1) memory and O(N) total time, versus attention’s growing cache and O(N^2) prefill.
For a CPU that means the bytes-per-token during decode stay flat no matter how long the conversation, which is precisely the property a memory-bound machine wants. The honest caveat is that pure SSMs can be weaker at exact recall — pulling a specific token from far back — because they compress history into a fixed state rather than keeping every key. The pragmatic answer, now common, is hybrid models that interleave a few full-attention layers among many SSM (or linear-attention) layers, keeping most of the constant-memory benefit while retaining sharp recall where it counts. For long-context on-device inference, this hybrid direction is one of the most promising things on the horizon.
CPU hardware I: AVX-512, VNNI, and AMX
CPUs did not stand still. The vector units widened and grew instructions aimed squarely at the low-precision integer math that quantized inference needs. AVX-512 processes 512-bit vectors — 64 INT8 values at once per lane. VNNI (Vector Neural Network Instructions) adds a fused multiply-accumulate over INT8 operands that does in one instruction what previously took several, roughly tripling INT8 dot-product throughput.
The bigger jump is Intel’s AMX (Advanced Matrix Extensions): a set of 2D tile registers with a dedicated matrix-multiply unit (TMUL) that computes a whole tile-by-tile INT8 or BF16 matmul per instruction — on the order of 1024 multiply-accumulates in a single op per core per cycle. That is a large multiplier on the CPU’s effective throughput for exactly the operation transformers are made of. The important framing, though: these units raise the CPU’s compute ceiling and its ridge point, which matters enormously for compute-bound prefill (digesting a long prompt in parallel). They do far less for memory-bound single-token decode, which remains gated by bandwidth. Great hardware, but pointed at the half of the problem that was never the bottleneck for generation.
CPU hardware II: caches and the bandwidth that actually gates you
Since decode is bandwidth-bound, the memory subsystem is the hardware that matters most, and it improved on two fronts. First, bandwidth: DDR5 roughly doubled per-channel throughput over DDR4, and dual-channel DDR5-5600 offers about 90 GB/s theoretical (call it 55–70 sustained). Apple’s M-series and other unified-memory designs go much further — the wide LPDDR5 buses on high-end parts reach several hundred GB/s, which is exactly why those machines punch so far above typical laptops at local LLM decode. Bandwidth maps almost linearly to tokens/s, so this is the number to watch.
Second, caches: last-level caches grew, and designs with a large stacked cache can keep a meaningful slice of a small quantized model on-die. When weights hit in cache instead of DRAM, effective bandwidth jumps several-fold and the roofline lifts. This is a real, if partial, escape from the DRAM wall — it helps the hottest, most-reused weights and the KV cache, though a multi-gigabyte model still overflows any cache and must stream from DRAM. The trajectory is encouraging: every generation adds bandwidth and cache, and unlike raw clock speed, both translate directly into on-device tokens/s.
CPU hardware III: on-device NPUs and heterogeneous inference
Recent laptop and phone SoCs ship a third compute block beside the CPU and GPU: a neural processing unit (NPU), a low-power accelerator built for INT8/INT4 matrix math, now commonly rated at 40+ TOPS. Its point is not raw peak — a discrete GPU still wins there — but performance per watt, which is what governs a fanless device running a model without draining the battery.
The realistic near-future is heterogeneous: split the workload across blocks by what each does best. The NPU or AMX-equipped CPU handles the compute-heavy, parallel prefill; the CPU with its fast path to full system RAM handles memory-bound decode, since the NPU’s own tight memory often cannot hold the whole model. Because prefill and decode have opposite bottlenecks — compute-bound versus bandwidth-bound — matching each phase to the block that suits it is a natural, and increasingly automated, division of labor. The strategic effect is that ‘the CPU’ is becoming a coordinated cluster of specialized units on one chip, and on-device inference the workload that finally exercises all of them together.
The economic and privacy case for on-device inference
Even where the cloud is faster, on-device inference wins on axes benchmarks miss. Economics: cloud inference is a metered per-token cost that recurs for the life of the product, and it scales with users; on-device inference runs on hardware the user already bought, so the marginal cost of a token is essentially zero and the provider carries no serving bill at all. For a feature invoked millions of times — autocomplete, summarization, on-the-fly translation — that difference dominates the unit economics.
Privacy and latency compound it. Data that never leaves the device cannot be logged, breached, or subpoenaed centrally, which is decisive for health, legal, financial, and enterprise contexts and increasingly for regulation. There is no network round trip, so first-token latency is governed by local compute, not a congested link, and the feature works offline — on a plane, in a dead zone, anywhere. These are not marginal preferences; they are hard product constraints that no amount of cloud speed satisfies. The forces in this article make on-device technically viable; this is why, once it is viable, it is often simply preferable.
A worked estimate: can a laptop run a useful assistant?
Put the pieces together for a concrete 2026-class laptop: a 7B model, over-trained and distilled so it is genuinely useful, quantized to 4 bits (about 3.7 GB), with GQA holding the KV cache small, on a machine sustaining ~65 GB/s. Decode speed by the formula: 65 / 3.7 ≈ 17.6 tokens/s. Comfortable reading speed is 5–8 tokens/s, so this clears ‘interactive’ with margin. Push the model to a 2-bit QAT build (~2 GB) and it is ~32 tokens/s; run it on an Apple-class unified-memory device at ~200 GB/s and even the 4-bit 7B hits ~50 tokens/s.
Prefill is the other half. Reading a 2,000-token prompt is compute-bound and parallel, so it leans on AVX-512/VNNI/AMX or the NPU; with AMX-class matrix units a few thousand prompt tokens digest in a second or two. The takeaway is quantitative and, for once, optimistic without hand-waving: the combination of sub-4-bit quantization, a smaller-but-stronger model, and DDR5-class bandwidth already lands a genuinely helpful assistant at faster-than-reading speed on hardware people own today — and every trend here is still moving in the favorable direction.
The honest limits: what the wall will not give back
Forward-looking should not mean credulous. The bandwidth wall is a wall, and some of it does not move. Single-stream decode can never exceed bandwidth / model_bytes; when the marketing says ‘runs a 70B model on your laptop,’ the arithmetic (65 / 35 ≈ 1.9 tokens/s at 4-bit) says it runs, but not usably. Quantization has a floor too — below ~2 bits, quality erosion becomes real, and BitNet-class ternary models must be trained that way from scratch, not conjured from an existing checkpoint.
Batching, the GPU’s trick for turning memory-bound decode compute-bound, barely helps a single local user with nothing to batch. Long context still inflates the KV cache and slows every step, GQA and SSMs notwithstanding. And a CPU SLM is still small: it will hallucinate more, reason less deeply, and know less than a frontier model, so the right architecture is often hybrid — the local model handles the private, latency-sensitive, high-volume majority and escalates the hard minority to the cloud. The future of CPU SLMs is not the death of the data center. It is a rebalancing, dictated by a bandwidth equation that rewards small, dense, well-trained models running next to the user — and that equation, unlike the hype, is one you can check yourself.