Every other article in this series zooms in on one gear — softmax, AdamW, the KV cache, an int4 kernel. This one is the assembly manual: how the gears mesh into a working machine. We take a single concrete artifact — a ~100M-parameter decoder-only model whose final home is a CPU — and walk it through the entire lifecycle: choosing the shape, sizing the data with a scaling law, tokenizing and packing, fixing the training config, training with checkpoints, measuring loss and perplexity, quantizing the weights, and finally serving tokens on commodity cores. At each stop we do the arithmetic that matters — parameters, tokens, FLOPs, gigabytes, tokens per second — and point to the sibling article that derives the piece in full. Read this end to end and the series stops being a pile of formulas and becomes a checklist you could actually execute. The numbers are illustrative but internally consistent; change the dimensions and the same relationships carry through.
The recipe at a glance
Building a small language model for CPU is a pipeline with nine stages, and each one constrains the next. Shape (how many parameters, arranged how) sets the data budget; the data budget sets the compute bill; the training config decides whether that compute converges; evaluation tells you when to stop; quantization shrinks the artifact; and the CPU runtime — threading, KV cache, SIMD — decides whether the finished model is usable or a slideshow.
The through-line is that decisions made at stage one echo all the way to stage nine. A wider model trains on more tokens, costs more FLOPs, produces a bigger file, reads more bytes per generated token, and therefore runs slower on a bandwidth-limited CPU. You cannot optimize any stage in isolation. So rather than treat these as independent topics, this recipe fixes one target — roughly 100M parameters, int4 on a laptop-class CPU — and lets every number fall out of that choice. Where a stage has its own dedicated derivation in this series, we give the headline result and a pointer, not a re-derivation. The goal here is the coherence between stages, which no single-topic article can show.
Step 1 -- pick the architecture and size
Start with a standard decoder-only transformer, because the whole series — from the complete transformer block to the multi-head attention math — is built on it. Our concrete shape: d_model = 768, n_layers = 12, n_heads = 12 (head dim 64), feed-forward d_ff = 3072 (the usual 4×), vocabulary 32,000, and a context length of 2,048.
Count the parameters, because this number drives everything downstream. Per layer, attention’s four projections cost 4 × d_model^2 = 4 × 768^2 ≈ 2.36M and the FFN costs 2 × d_model × d_ff ≈ 4.72M, so a layer is about 7.08M; twelve layers give ~85M non-embedding parameters. The tied token-embedding / output matrix (see the tied-embeddings piece) adds 32000 × 768 ≈ 24.6M, for a total near 110M — call it a ~100M model. Norm and bias terms are rounding error. Choices like RMSNorm over LayerNorm, SwiGLU over GELU, RoPE for position, and grouped-query attention change the constant factors and the inference profile, but not this ballpark. The SLM-architectures-compared article weighs those trade-offs; here the takeaway is simply N ≈ 110M.
Step 2 -- set the token budget with Chinchilla
How much data does a 110M model want? The Chinchilla compute-optimal result gives a clean rule of thumb: train on roughly 20 tokens per parameter. That puts our budget at 110M × 20 ≈ 2.2 billion tokens. The Chinchilla article derives why the optimum balances parameters and data rather than pouring everything into size; the operational consequence is that a 110M model trained on 200M tokens is badly undertrained, and one trained on 50B tokens is spending compute you could have used to grow the model.
Two caveats sharpen the estimate. First, Chinchilla optimizes training compute; for a model you will run millions of times on a CPU, it is often worth over-training past 20× — spending more data to get a smaller, cheaper-to-serve model, exactly the inference-optimal reasoning the SLM-scaling discussions favor. Many strong small models see 20–40× or more. Second, tokens are not words: with a 32k BPE vocabulary, 2.2B tokens is on the order of 1.5–1.7B words of text. We will treat D ≈ 2.2B tokens as the baseline and revisit over-training when we count the serving cost.
Step 3 -- prepare and tokenize the data
Raw text becomes training-ready in three moves, each with its own article. Curate: the training-data-for-SLM piece argues that at this scale data quality dominates quantity — dedup, filter boilerplate, balance the domain mix — because a 110M model has little capacity to waste memorizing junk. Tokenize: train a byte-pair-encoding vocabulary (the tokenizer-math article) sized to 32,000, a sweet spot that keeps the embedding table — a fifth of our parameters — from dominating while still compressing text to roughly 0.75 words per token.
Pack: the dataloader-and-tokenization pipeline concatenates documents into one long token stream and slices it into fixed 2048-token windows, separated by a document/EOS token so the model learns boundaries. Packing matters because padding wastes FLOPs on nothing; at 2.2B tokens and 2,048 per window, you have about 1.07M training sequences. Store them as a memory-mapped array of uint16 token IDs (32k fits in 16 bits): 2.2e9 × 2 bytes ≈ 4.4 GB on disk, streamed lazily so RAM is never the bottleneck. The loader shuffles at the window level and yields batches; determinism plus a saved shuffle seed makes runs reproducible.
Step 4 -- the training config: batch, LR, schedule, precision
This is where four sibling articles converge into a handful of numbers. Batch: aim for roughly 0.5M tokens of gradient signal per step — large enough for a stable estimate (see the batch-size math). A CPU or a modest GPU cannot hold that in one shot, so use a micro-batch of 16 sequences (16 × 2048 = 32,768 tokens) and gradient accumulation of 16 to reach ~524k tokens per optimizer step (the microbatch article shows why the accumulated gradient equals the big-batch gradient). At 2.2B tokens that is about 4,200 optimizer steps.
Learning rate and schedule: AdamW with a peak LR of 3e-4, a short linear warmup of ~200 steps, then cosine decay to 3e-5 — the shape the learning-rate-schedules article recommends, warmup to survive the unstable start, cosine to anneal into a good minimum. Precision: train in bf16 mixed precision (the mixed-precision article) — bf16 compute with an fp32 master copy of the weights, which halves memory traffic and, unlike fp16, needs no loss scaling because bf16 keeps fp32’s exponent range. Weight init follows the weight-initialization article’s scaled-residual scheme so activations don’t explode at depth 12.
Step 5 -- the optimizer and stability
The optimizer is AdamW, and the SGD-vs-Adam article explains why: at this scale the per-parameter adaptive step sizes and decoupled weight decay (0.1) converge far more reliably than plain SGD, for a memory price we can afford. Typical betas are β_1 = 0.9, β_2 = 0.95 — the slightly lower second moment is common for language models. That memory price is concrete: Adam keeps two state tensors (first and second moment) per parameter, so on top of the weights you carry 2 × N extra values.
Stability is the other job. The gradient-clipping-and-stability article prescribes clipping the global gradient norm to 1.0 before each step, which caps the damage from the occasional bad batch that would otherwise spike the loss and corrupt the Adam moments. Together, warmup, cosine decay, bf16’s wide range, scaled initialization, and gradient clipping form a stability stack: remove any one and a 12-layer model trained on billions of tokens will, sooner or later, diverge. When it does, the loss-curves-diagnosis article is the field guide — a sudden spike, a plateau, or a slow drift each point to a different culprit in this list.
Step 6 -- train with checkpointing
Two different things share the name ‘checkpoint,’ and you need both. Activation checkpointing (a.k.a. gradient checkpointing) trades compute for memory during the backward pass: instead of storing every layer’s activations, you keep a few and recompute the rest, cutting activation memory sharply at the cost of one extra forward pass. On a memory-tight box that is what lets the batch fit at all.
State checkpointing is the save-your-work kind: periodically write the model weights, the Adam moments, the LR-schedule position, the dataloader offset, and the RNG state to disk. Save the full optimizer state, not just weights, or you cannot resume without a loss discontinuity — the moments would restart cold. For our run, a full checkpoint is about the training-memory footprint below (a few GB); writing one every, say, 500 steps costs a little disk and buys crash recovery for a job that may run for days. Total training memory in bf16 mixed precision is roughly 16 bytes × N: 2 (bf16 weights) + 2 (bf16 grads) + 4 (fp32 master) + 4 + 4 (Adam moments) = 16 × 110M ≈ 1.76 GB, plus activations, so plan for ~2–4 GB.
Step 7 -- the compute bill and an honest wall-clock
The single most useful training estimate is C ≈ 6 · N · D FLOPs — six FLOPs per parameter per token, covering the forward and backward passes. For us: 6 × 110e6 × 2.2e9 ≈ 1.45e18 FLOPs, about 1.5 ExaFLOPs. That number, divided by your hardware’s sustained throughput, is the wall-clock.
Here honesty matters. A modern multi-core CPU with AVX-512 might sustain on the order of 100 GFLOP/s on this workload, giving 1.45e18 / 1e11 ≈ 1.45e7 s ≈ 168 days. Pre-training a 110M model on a CPU is, bluntly, impractical. A single mid-range GPU at ~100 TFLOP/s effective does the same run in ~4 hours. So the recipe splits cleanly: the config above is hardware-agnostic, but you rent a GPU for pre-training (or fine-tune a smaller run, where CPU is tolerable). The CPU is the deployment target — the whole premise of this series — not the training target. Everything from Step 8 onward is about making the trained artifact run well on the CPU it will actually live on.
Step 8 -- evaluate with loss and perplexity
The training objective is next-token cross-entropy, derived in its own article as the negative log-likelihood the model assigns to the true next token. You watch it on a held-out validation split, never the training stream, so you are measuring generalization rather than memorization. A well-trained 110M model on general English lands somewhere around a validation loss of ~3.0–3.3 nats per token, depending heavily on the data.
Perplexity is just that loss exponentiated: PPL = exp(loss), the effective number of equally-likely choices the model is deciding among at each step (the perplexity-and-evaluation article makes this precise). A loss of 3.1 is a perplexity of about exp(3.1) ≈ 22. Lower is better; the absolute value is only comparable across models that share a tokenizer, since perplexity is per-token and tokenization changes what a token is. Perplexity is a proxy, not the goal — it correlates with but does not guarantee downstream quality — so pair it with a few task benchmarks. Track the curve: a validation loss that stops falling while training loss keeps dropping is the loss-curves signature of overfitting, and your cue to stop.
Step 9 -- quantize for inference
The trained weights are fp32: 110M × 4 bytes ≈ 440 MB. That is both too big and, on a CPU, too slow — every generated token must read the weights from RAM, and decode is bandwidth-bound, so bytes are time. Quantization is the highest-leverage inference optimization we have.
Post-training quantization to int8 roughly halves-and-halves-again the footprint to ~110 MB, usually with negligible quality loss. int4 (as in the GGUF/GPTQ/AWQ family) reaches ~55 MB — an 8× shrink from fp32 — at a small, measurable perplexity cost, which the quant-evaluation approach quantifies so you pick a bit-width on evidence, not vibes. The mechanics live in the quantization-layouts and weight-storage articles: weights are stored in small blocks, each with its own scale (and maybe zero-point), so a block of int4 values dequantizes on the fly inside the matmul kernel. You quantize the big weight matrices; norms and the occasional sensitive layer often stay in higher precision. The result is a ~55 MB file that fits in cache-friendly working sets and, crucially, reads 8× fewer bytes per token than fp32.
Step 10 -- the CPU inference pipeline
Now deploy. Generation is autoregressive (the generation-loop article): a prefill pass consumes the prompt in parallel, then decode emits one token at a time, each feeding back as input. These two phases have opposite bottlenecks, and the CPU runtime must respect both.
Prefill is compute-bound: it is a big batched matrix multiply over all prompt positions, so it wants raw FLOPs — which on a CPU means good matmul kernels and SIMD. The cpu-matmul-kernels article covers blocking the multiply to fit the cache hierarchy; the SIMD-for-transformers article covers vectorizing the inner loop so one instruction does 8 or 16 multiply-adds (AVX2/AVX-512), including dequantizing int4 blocks in registers. The cpu-cache-hierarchy article explains why tiling to L1/L2 is what separates a fast kernel from a slow one — a matmul that thrashes DRAM leaves most of the CPU idle. Decode is memory-bandwidth-bound: each step touches every weight once but does little arithmetic, so it is limited by how fast you can stream the (now int4) weights from RAM. That single fact governs the throughput we compute next.
Step 11 -- threading, KV cache, and throughput
The KV cache is what makes decode affordable: instead of re-attending over the whole prefix every step, you cache each layer’s keys and values and only compute the new token’s (the KV-cache-math article). Its size is 2 × n_layers × d_model × bytes per token — here 2 × 12 × 768 × 2 (fp16) ≈ 36 KB/token, so a full 2,048-token context holds about 74 MB of cache. Grouped-query attention or KV-cache quantization shrink that further; at 110M it is manageable as-is.
Throughput, decode side: each token streams the ~55 MB int4 weight set once. On a CPU with ~50 GB/s of usable memory bandwidth the ceiling is 50e9 / 55e6 ≈ 900 tokens/s. Real memory-bandwidth utilization on a decode loop is more like 20–35%, so expect ~150–300 tokens/s — comfortably interactive for a 110M model. Threading helps prefill (compute-bound, scales with cores) far more than decode (bandwidth-bound, saturates after a few cores because the bottleneck is the memory bus, not the ALUs). The cpu-inference-pipelines article ties these together: pin threads, size the thread pool to the memory system rather than the core count, and keep the working set cache-resident.
Step 12 -- put the numbers on one page
The value of a capstone is seeing the whole budget at once. For our ~110M model, context 2,048, int4 on a laptop-class CPU:
| Quantity | Value | Where it comes from |
|---|---|---|
| Parameters (N) | ~110M | 12 layers × 7.08M + 24.6M tied embed |
| Token budget (D) | ~2.2B | Chinchilla, 20× N |
| Training FLOPs | ~1.45e18 | 6·N·D |
| Training memory | ~2–4 GB | 16 bytes/param (bf16 + AdamW) + activations |
| Train time (1 GPU) | ~4 hours | @ ~100 TFLOP/s effective |
| Weights fp32 / int8 / int4 | 440 / 110 / 55 MB | N × 4 / 1 / 0.5 bytes |
| KV cache @ 2048 | ~74 MB | 36 KB/token × 2048 |
| Decode throughput | ~150–300 tok/s | 50 GB/s ÷ 55 MB, 20–35% MBU |
Read the table top to bottom and the coupling is obvious: N sets D, D sets the FLOP bill, N sets the file size, the file size sets the decode speed. Halve d_model and every row moves together. This is why the series is a series and not a grab-bag — the same handful of dimensions propagate through every stage.
Common pitfalls and where it breaks
Watch the joints between stages, because that is where recipes fail. Undertraining: skimping on tokens to save GPU hours leaves capacity on the table; for a model you will serve endlessly, over-training past Chinchilla is usually the better trade. Tokenizer / eval mismatch: perplexity is only comparable across a shared vocabulary — changing the tokenizer silently changes the number. Quantizing blind: dropping to int4 without measuring perplexity can quietly cost real quality; always run the quant-evaluation step. Over-threading decode: throwing all cores at a bandwidth-bound loop buys nothing and can hurt via contention — the memory bus, not the core count, is the ceiling.
The deepest pitfall is treating the stages as independent. A wider model for ‘better quality’ also means more tokens to train it right, a bigger file, more bytes per token, and a slower CPU decode — a quality win that silently taxes every other row of the table. The discipline the whole series teaches is to reason about the artifact as one coupled system: pick the shape with deployment in mind, size the data to the shape, and let evaluation — not optimism — tell you when each stage is done.