Open the config files of GPT-2, Llama, Mistral, Phi, Gemma, and Qwen side by side and you will see the same skeleton: a stack of decoder blocks, each with self-attention and a feed-forward network, wrapped in residual connections. What separates them is a short list of decisions made at every block — how the heads share keys and values, which normalizer sits where, how position is encoded, which activation gates the FFN, whether the input and output embeddings are the same matrix, and how big the vocabulary is. Individually each choice looks like a footnote. Together they decide the parameter budget, the KV-cache footprint, the training stability, and whether the model runs on a CPU at a usable speed. This piece walks the axes one at a time, does the arithmetic that makes each trade-off concrete, lays the six families out in a single table, and closes with a worked parameter budget so you can read any model card and know where its weights went.
Why the axes matter more at small scale
At 175B parameters, architecture choices are largely a wash — the model is so overparameterized that a slightly worse normalizer or a wasteful FFN just gets absorbed. At 1–7B parameters, running on a laptop CPU or a single consumer GPU, every choice is visible in the benchmark and in the latency. A small model has no slack: its parameter budget is tight, its KV cache competes with everything else in RAM, and its training run is short enough that instability is fatal rather than annoying.
That is why the small-model era — Llama, Mistral, Phi, Gemma, Qwen — converged on a specific recipe that GPT-2 did not use, and why the few families that deviate (Phi keeping full multi-head attention, Gemma carrying a 256K vocab) make those deviations deliberately. The rule of thumb for the rest of this article: a choice is worth making when it improves quality per parameter, quality per FLOP, or quality per byte of KV cache — the three currencies a small model is always short of. Read every axis below as an answer to “which of those three does this buy, and what does it cost in the others?”
Depth vs width: the aspect ratio
Given a fixed parameter budget you can spend it on more layers (depth, n_layers) or wider layers (width, d_model). The parameter count is dominated by terms that scale like n_layers × d_model^2, so many (depth, width) pairs hit the same budget: a deep-thin model and a shallow-wide model can weigh the same.
They do not behave the same. Depth adds sequential composition — more layers means more stages of reasoning, which tends to help on tasks that need multi-step transformation, but it lengthens the critical path: layer k cannot start until layer k-1 finishes, so depth costs latency and is harder to parallelize. Width adds representational capacity per step and is highly parallel (bigger matmuls saturate hardware better), but attention and FFN cost grows with d_model^2. Compare Qwen’s 3B (d_model=2048, 36 layers — deep and thin) with Phi-3-mini (d_model=3072, 32 layers — wider). The deep-thin design leans on composition and keeps each matmul cheap; the wide design leans on per-token capacity and better hardware utilization. Neither wins universally — it is a genuine dial, and CPU inference often prefers moderate width so the matmuls stay cache-friendly.
Attention I: MHA, and what the KV cache costs
Classic multi-head attention (MHA) gives every query head its own key and value head. With h heads of dimension d_head and d_model = h × d_head, the four projections W_Q, W_K, W_V, W_O are each [d_model, d_model], so attention costs 4 × d_model^2 parameters per layer.
The parameters are not the pain point — the KV cache is. During generation you must store the keys and values of every past token so you do not recompute them. The cache size per token is 2 × n_layers × h × d_head values (the 2 is K and V). For a 24-layer model with 16 heads of dimension 128 that is 2 × 24 × 16 × 128 = 98,304 values per token, or about 192 KiB at fp16. At an 8K context that is 8192 × 192 KiB ≈ 1.6 GB — for one sequence. On a CPU SLM where RAM is the budget, that cache often dwarfs the activations and rivals the weights. This single number is why the next two attention variants exist: they attack the KV cache directly.
Attention II: MQA and GQA — sharing keys and values
Multi-query attention (MQA) keeps h query heads but collapses the keys and values to a single shared head. The KV cache shrinks by a factor of h — from 2 × n_layers × h × d_head to 2 × n_layers × d_head. That is a huge win for memory-bound decode, but sharing one KV head across all queries can cost quality and destabilize training at scale.
Grouped-query attention (GQA) is the compromise that essentially every modern small model adopts. Split the query heads into g groups; each group shares one KV head, so you have g KV heads with 1 ≤ g ≤ h. GQA interpolates smoothly: g = h is MHA, g = 1 is MQA. The cache scales as 2 × n_layers × g × d_head. Take the 24-layer example with 16 query heads but g = 4 KV heads: the per-token cache drops from 98,304 to 2 × 24 × 4 × 128 = 24,576 values — a 4× reduction, taking the 8K-context cache from ~1.6 GB to ~400 MB with negligible quality loss. Qwen 2.5-3B uses 8× GQA (16 query, 2 KV heads); Mistral and Llama use 4×. It is the highest-leverage KV optimization in the small-model playbook.
Normalization I: LayerNorm vs RMSNorm
Every sub-block is normalized to keep activations well-scaled. LayerNorm (GPT-2) subtracts the mean and divides by the standard deviation across the feature dimension, then applies a learned scale γ and bias β: y = γ · (x - μ) / √(σ^2 + ε) + β. It has 2 × d_model parameters per norm and computes a mean, a variance, a subtraction, and a division.
RMSNorm drops the mean-centering and the bias entirely, normalizing only by the root-mean-square of the activations: y = γ · x / √(mean(x^2) + ε). It has just d_model parameters per norm and skips the mean and the subtraction. Empirically the re-centering LayerNorm does turns out to be nearly irrelevant to model quality, so RMSNorm keeps the benefit (stable scale) at lower cost. That is why every post-GPT-2 family in our table — Llama, Mistral, Phi, Gemma, Qwen — switched to RMSNorm. The saving per norm is tiny in parameters, but on a CPU the removed mean and subtraction over millions of activations per token is real, and the simpler op vectorizes cleanly.
Normalization II: pre-norm, post-norm, and sandwich
Where the norm sits matters as much as which norm it is. The original Transformer used post-norm: x + Sublayer(x) then normalize. This puts the normalizer outside the residual path, which makes deep stacks hard to train — gradients must survive many un-normalized residual additions, so post-norm networks need careful warmup and often diverge when deep.
Pre-norm moves the normalizer inside the residual branch: x + Sublayer(Norm(x)). Now there is a clean identity path from input to output with nothing on it, so gradients flow freely and deep models train stably with minimal warmup. Pre-norm is the default for essentially every modern small model. Gemma-2 goes further with a sandwich: it normalizes both the input and the output of each sub-block (pre- and post-norm together), buying extra stability at the cost of two norms per sub-block instead of one. The progression — post-norm → pre-norm → sandwich — is a steady trade of a little compute for a lot of training robustness, which matters most for small models whose training runs are too short to babysit an unstable loss curve.
Activation: GELU vs SwiGLU and the FFN budget
The feed-forward network is where most of a model’s parameters live. A classic FFN is two matrices with a nonlinearity between: FFN(x) = W_2 · φ(W_1 x), with W_1: [d_model, d_ff] and W_2: [d_ff, d_model], costing 2 × d_model × d_ff parameters. GPT-2 uses φ = GELU and d_ff = 4 × d_model.
Modern models use SwiGLU, a gated FFN with three matrices: FFN(x) = W_down · (SiLU(W_gate x) ⊙ W_up x), where ⊙ is elementwise product and SiLU(z) = z · σ(z). The extra matrix means three weight tensors of size d_model × d_ff, i.e. 3 × d_model × d_ff parameters. To keep the budget the same as a 4d GELU FFN, gated models shrink the hidden width to d_ff ≈ (8/3) × d_model — hence the odd-looking values like Llama’s d_ff = 5504 for d_model = 2048. The gating lets the network modulate its own activations multiplicatively, which reliably beats a plain GELU FFN at equal parameter count. Gemma uses the GELU-flavored cousin GeGLU; the gating idea is the same.
Positional scheme: learned vs RoPE vs ALiBi
Attention is permutation-invariant, so position must be injected. GPT-2 used a learned absolute embedding: a lookup table of [max_len, d_model] added to the token embeddings. It works but cannot extrapolate past max_len — positions it never saw in training have no vector — and it spends parameters on the table.
RoPE (rotary position embedding) is the modern default. It encodes position by rotating the query and key vectors by an angle proportional to their position, so the dot product q_m · k_n depends only on the relative offset m - n. It adds zero parameters, is relative by construction, and — crucially — can be stretched to longer contexts after training by scaling the rotation frequencies (NTK / YaRN / LongRope, as Phi-3 uses). ALiBi takes a different route: no embedding at all, just a linear distance penalty added to the attention scores, which biases each head toward nearby tokens and extrapolates gracefully. RoPE won the small-model consensus — Llama, Mistral, Phi, Gemma, and Qwen all use it — because it is parameter-free, relative, and length-extensible, exactly the properties a small long-context model needs.
Tied embeddings: reusing the vocabulary matrix
A decoder has two big vocabulary matrices: the input embedding [vocab, d_model] that maps token IDs to vectors, and the output projection [d_model, vocab] that maps the final hidden state to logits. Weight tying makes them the same matrix (transposed), halving the vocabulary parameter cost from 2 × vocab × d_model to vocab × d_model.
The size of that saving depends entirely on the vocab. For Gemma with vocab = 256,000 and d_model = 2304, one embedding matrix is 256000 × 2304 ≈ 590 million parameters — tying saves a second 590M, which for a 2B model is enormous, so Gemma ties. Small Qwen models tie for the same reason. Llama, with a modest 32K vocab, leaves them untied because the saving is smaller relative to a 7B budget and untied output heads can slightly help quality. The decision rule is a ratio: tying matters exactly when vocab × d_model is a large fraction of the total budget — which is precisely the regime of small models with large vocabularies. This is the axis where the vocab choice and the tying choice interact most sharply.
Vocab size: the tokenizer's footprint
Vocabulary size is a quiet but consequential axis. A larger vocab means each token carries more text on average, so a fixed sequence covers more content — fewer tokens per document lowers the quadratic attention cost and speeds generation. It also improves coverage of non-English scripts and code, where a small English-centric vocab shatters words into many sub-tokens. That is why Qwen ships ~152K tokens for strong multilingual and Chinese coverage, and Gemma ships 256K.
The cost is parameters and compute concentrated in two places: the embedding table and the final softmax, both vocab × d_model. At GPT-2’s 50K vocab this was minor; at 256K it is a large chunk of a 2B model’s weights (hence Gemma’s tying). The softmax over a 256K vocabulary is also a real per-token cost on a CPU. So vocab size trades sequence efficiency and language coverage against parameter and softmax cost. Small English-focused models keep it near 32K (Llama, Mistral, Phi); multilingual models pay for a big vocab and then recover the parameters by tying. There is no free lunch — only a choice about where the tokens should be spent.
Dense vs MoE: activating a fraction of the weights
Everything so far assumed a dense model: every parameter participates in every token. A Mixture-of-Experts (MoE) FFN instead holds many parallel expert FFNs and a small router that, per token, selects the top-k experts (typically 2 of 8). Only those k run, so the model has a large total parameter count but a much smaller active count per token.
The math is the whole appeal: Mixtral 8×7B holds ~47B total parameters but activates only ~13B per token, so it computes like a 13B model while storing the knowledge of a much larger one. For a compute-constrained setting that is a great trade. For a memory-constrained CPU SLM it is often the wrong trade: you must hold all 47B parameters in RAM even though each token touches a fraction, and the router adds load-balancing complexity and can hurt latency with sparse memory access. That is why most pure small models — Phi, Gemma, small Qwen, small Llama — stay dense, and MoE shows up mainly when you have GPU memory to spare and want capacity without proportional FLOPs. Dense trades capacity for a tight memory footprint; MoE trades memory footprint for cheap capacity.
The families, side by side
Here are the six families along the axes above. The striking thing is how much they agree: after GPT-2, the column values barely move — the recipe is RMSNorm + pre-norm + RoPE + gated FFN + GQA, and the real differentiation is in vocab, tying, and a few signature tricks.
| Family | Attention | Norm | Position | Activation | Tied emb | Vocab | Signature |
|---|---|---|---|---|---|---|---|
| GPT-2 | MHA | LayerNorm, pre | Learned abs | GELU | Yes | 50K | The baseline everyone moved off |
| Llama 2/3 | GQA (4×) | RMSNorm, pre | RoPE | SwiGLU | No | 32K → 128K | The reference recipe |
| Mistral | GQA (4×) | RMSNorm, pre | RoPE | SwiGLU | No | 32K | Sliding-window attention |
| Phi-3 | MHA | RMSNorm, pre | RoPE (LongRope) | SwiGLU | No | 32K | Synthetic-data quality; keeps MHA |
| Gemma 2 | GQA | RMSNorm, sandwich | RoPE | GeGLU | Yes | 256K | Huge vocab; local/global attn |
| Qwen 2.5 | GQA (8×) | RMSNorm, pre | RoPE | SwiGLU | Yes (small) | 152K | QKV bias; multilingual |
Read across a row and you can predict the model’s memory and latency profile before you download a single weight. Read down a column and you see the field’s consensus — and the deliberate exceptions.
A worked parameter budget
Let us build a ~1.1B model and account for every weight, so you can do this for any config. Take d_model = 2048, n_layers = 24, 16 query heads of d_head = 128, GQA with 4 KV heads, SwiGLU with d_ff = 5504, vocab = 32,000, tied embeddings.
Embedding (tied): vocab * d_model = 32000 * 2048 = 65.5M
Per layer:
Attn Q, O: 2 * d_model^2 = 2 * 2048^2 = 8.39M
Attn K, V: 2 * d_model * (4 * d_head)
= 2 * 2048 * 512 = 2.10M
SwiGLU: 3 * d_model * d_ff = 3 * 2048 * 5504 = 33.8M
RMSNorm x2: 2 * d_model = ~0.004M
------------------------------------------------------------
per-layer total = 44.3M
All layers: 24 * 44.3M = 1063M
Embedding: = 65.5M
Final norm: = ~0.002M
------------------------------------------------------------
TOTAL = ~1.13BTwo lessons jump out. First, the FFN dominates: 33.8M of each layer’s 44.3M — over 75% — is SwiGLU. Second, GQA barely dents the parameter count (K and V are only 2.1M of 44.3M) — its payoff is the KV cache, not the weights. Had we used MHA, K and V would quadruple to 8.4M, adding ~150M parameters and quadrupling the runtime cache. The budget makes the priorities obvious.
Reading a model card, and choosing for CPU
Put the axes together and a model card stops being a wall of numbers. d_model and n_layers tell you the aspect ratio and the bulk of the weights (mostly FFN). The ratio of query heads to KV heads tells you the GQA factor and thus the KV-cache footprint — the number that decides whether a long context fits in your RAM. The activation and d_ff tell you if it is a gated FFN and confirm the (8/3)d tell. The vocab and a tied-embedding flag tell you how much of the budget is the vocabulary matrix.
For a CPU SLM specifically, the priorities invert from the datacenter. Memory bandwidth, not FLOPs, is the bottleneck at decode, so aggressive GQA (or even MQA) is worth more than another benchmark point, and a dense model is usually better than MoE because you cannot afford to hold unused experts in RAM. Moderate width keeps matmuls cache-friendly; RMSNorm and RoPE cost nothing and extend context for free. The winning small-CPU recipe is almost exactly the modern consensus — deep-ish and thin, GQA, RMSNorm pre-norm, RoPE, SwiGLU, dense, tied embeddings if the vocab is large — because that recipe was tuned under the same three scarcities: parameters, FLOPs, and bytes of cache.