You have met the parts separately — attention mixes information across positions, the feed-forward network transforms each position on its own, normalization keeps activations well-scaled, and residual connections let gradients flow. This article is where they click together into the single repeating unit that a transformer stacks dozens of times: the block. The modern recipe is compact and worth memorizing: x → x + Attn(Norm(x)) → x + FFN(Norm(x)). Two sublayers, each wrapped in a norm-then-add sandwich, sharing one residual stream that runs unbroken from the embedding to the output. Once you can trace a tensor through that pattern — naming its shape at every step — and once you can count the block’s parameters and its per-token FLOPs on the back of an envelope, the whole model stops being a mystery and becomes arithmetic. We will build the block, count it, cost it, stack it, wrap it with the embedding and LM head, and close with a fully worked budget for a small concrete configuration.
The block is the unit of the model
A transformer is not really a single monolithic network; it is one block repeated L times, bracketed by an embedding at the bottom and a language-model head at the top. Every block has the identical shape and the identical parameter budget (only the learned weights differ), which is exactly why the model scales so cleanly: to make it bigger you either widen the block (raise the model dimension d) or add more blocks (raise L).
That regularity is a gift for reasoning about cost. If you understand one block — its data flow, its shapes, its parameter count, its FLOPs — you understand the model up to a multiply by L plus two boundary pieces. This piece is deliberately integrative: the internals of attention, the FFN, and normalization each have their own article in this series, so here we treat them as known components and focus on how they compose. The interesting questions at this altitude are not ‘what is softmax’ but ‘in what order do the pieces run, what shape is the tensor at each hand-off, how many parameters does the whole thing hold, and how much compute does one token cost as it passes through.’
The pre-norm data flow, exactly
A modern block contains two sublayers — a self-attention sublayer and a position-wise feed-forward sublayer — and each is wrapped in the same three-step pattern: normalize, transform, add. Written out for input x (shape [B, N, d]):
h = x + Attn(Norm(x)) # attention sublayer
y = h + FFN(Norm(h)) # feed-forward sublayerRead it carefully. The input to each sublayer is first passed through a normalization; the normalized copy is fed to the transform (attention or FFN); and the transform’s output is added back to the un-normalized input. The norm is inside the residual branch, not on the main path — that is the defining feature of a pre-norm block. The main path, the thing the additions accumulate onto, is never itself normalized; it flows straight through. Each sublayer therefore reads a cleaned-up view of the stream and writes a correction back onto it. Two sublayers, two norms, two additions — that is the entire block, and every transformer you have heard of is this pattern repeated.
Why pre-norm, not post-norm
The original 2017 transformer put the norm after the residual add (Norm(x + Sublayer(x))) — ‘post-norm’. It works, but it is temperamental to train deep: the residual path gets re-normalized at every layer, so signal and gradient magnitudes can swing, and deep post-norm stacks usually need a learning-rate warm-up and careful initialization to converge at all.
Pre-norm — x + Sublayer(Norm(x)) — moves the norm onto the branch and leaves the residual highway untouched. Because the additions land on a path that is never rescaled, there is a clean identity connection from the embedding all the way to the output: gradients can flow straight down it without being repeatedly squashed. That single change is what made it practical to train blocks 24, 48, or 96 deep, and it is why essentially every current large model (GPT-style, LLaMA-style) is pre-norm. The cost is subtle — the residual stream’s magnitude tends to grow with depth since nothing normalizes the main path — which is precisely why a final normalization sits at the very top of the stack, just before the head, to tidy up before the projection to logits.
The residual stream is the backbone
The most useful mental model of a transformer is not ‘a pipeline of layers’ but a single residual stream: a running vector of width d at each position that every sublayer reads from and writes back to by addition. The embedding initializes the stream; each attention and FFN sublayer adds its contribution; the head reads the final state. Because the operations are additive, the stream behaves like a shared communication bus — a place different layers deposit and retrieve information.
This framing pays off immediately for shapes and cost. The stream is [B, N, d] from the embedding output to the final norm input — it never changes width. Attention may explode internally into [B, h, N, N] score matrices and the FFN may balloon to [B, N, d_ff] in its hidden layer, but both project back to d before adding to the stream. That invariant — every sublayer is a function from d to d whose output is added, not concatenated — is what lets you stack blocks arbitrarily and what makes the parameter and FLOP arithmetic so regular.
Sublayer one: attention, as a component
Inside the first sublayer, self-attention lets every position gather information from every other position. As a black box it maps the normalized stream [B, N, d] to an output [B, N, d] using four learned projections. Queries, keys, and values are formed by Q = X·W_Q, K = X·W_K, V = X·W_V, each W of shape [d, d]; the result is split across h heads of width d_head = d/h, attention is computed per head, the heads are concatenated back to width d, and a final output projection W_O: [d, d] mixes them.
For the block-level view, two facts matter. First, the parameters are the four d×d matrices — 4·d^2 weights, independent of sequence length. Second, the compute has two parts: the projections (linear in N, captured by the 2×params rule below) and the score/aggregate step QK^T then ·V (quadratic in N). The internals — softmax, scaling by √d_head, the KV cache — are covered in the attention article; here we only need its shape signature and its two cost terms.
Sublayer two: the feed-forward network
The second sublayer is a small two-layer MLP applied independently to each position — there is no mixing across positions here; all the cross-token communication happened in attention. It expands the stream to a wider hidden dimension d_ff (classically 4·d), applies a nonlinearity, and projects back:
FFN(x) = W2 · act(W1 · x + b1) + b2
W1: [d, d_ff] W2: [d_ff, d] act = GELU / SwiGLU / ReLUShape-wise the hidden activation is [B, N, d_ff] before it collapses back to [B, N, d] for the residual add. The parameters are the two big matrices: d·d_ff + d_ff·d = 2·d·d_ff, which at d_ff = 4d is 8·d^2 — twice the attention block’s weights. That is the quiet headline of the whole model: the FFN, not attention, holds most of the parameters and, at ordinary context lengths, does most of the compute. Gated variants like SwiGLU add a third matrix and usually shrink d_ff to roughly (8/3)·d to keep the count similar, but the ‘FFN dominates the weights’ conclusion is unchanged.
The two normalizations
Each sublayer’s norm standardizes the stream before the transform reads it, so the input to attention and to the FFN always has a controlled scale regardless of how large the residual stream has grown. The original transformer used LayerNorm (subtract the mean, divide by the standard deviation over the feature axis, then scale by a learned γ and shift by a learned β); most recent models use RMSNorm, which drops the mean-centering and the β, dividing only by the root-mean-square and scaling by γ.
The cost accounting here is tiny but worth stating precisely. A LayerNorm holds 2·d parameters (γ and β); an RMSNorm holds d (γ only). With two norms per block that is 4·d or 2·d parameters — utterly negligible next to the 12·d^2 of the sublayers (for d = 512, that is 2,048 versus ~3.1 million). Their FLOPs are similarly a rounding error. Norms matter enormously for trainability and almost not at all for the parameter and FLOP budget — a useful thing to remember when someone worries about ‘the cost of all those norms.’
Shapes at every step
Here is the full trace for one block, batch size B, sequence length N, model width d, h heads of width d_head = d/h, hidden width d_ff:
x [B, N, d] # residual stream in
Norm(x) [B, N, d]
Q,K,V = xW_Q,xW_K,xW_V [B, N, d] each
reshape to heads [B, h, N, d_head]
scores = QK^T / √d_head [B, h, N, N] # the quadratic term
softmax(scores) [B, h, N, N]
· V (aggregate) [B, h, N, d_head]
concat heads [B, N, d]
· W_O (output proj) [B, N, d]
h = x + attn_out [B, N, d] # residual add #1
Norm(h) [B, N, d]
· W1 + act [B, N, d_ff] # FFN expand
· W2 [B, N, d] # FFN contract
y = h + ffn_out [B, N, d] # residual add #2, block outNotice the two places the tensor leaves width d — the [B, h, N, N] score matrices and the [B, N, d_ff] FFN hidden — and how both return to [B, N, d] before an add. The block is width-preserving end to end, which is exactly what lets the next block consume it unchanged.
Counting the block’s parameters
Add up the learned weights, ignoring the negligible norms and (optional) biases:
Attention: W_Q,W_K,W_V,W_O = 4 · d^2
FFN: W1 + W2 = 2 · d · d_ff = 8 · d^2 (at d_ff = 4d)
Norms: 2 × (2d or d) = 4d or 2d (negligible)
-----------------------------------------------------------
Per block ≈ 4d^2 + 8d^2 = 12 · d^2The 12·d^2 per block rule of thumb is one of the most useful numbers in the field. It says a block’s size grows with the square of the width and that the split is fixed at one-third attention, two-thirds FFN. It also makes clear why widening a model is so expensive: doubling d quadruples every block’s parameters. The approximation quietly drops biases and norm scales — correct to well under a percent — and assumes the standard 4× FFN ratio; a gated SwiGLU FFN with three matrices and d_ff ≈ (8/3)d lands in the same neighborhood. For estimation you almost never need more precision than 12·d^2.
From block to whole model
Multiply the block by depth and add the two boundary pieces. The stack is L identical blocks: L · 12 · d^2 weights, the non-embedding parameter count — the number people mean when they say a model ‘has N parameters’ for compute purposes. Around it:
Token embedding: V · d # lookup table, vocab V
(blocks) L · 12 · d^2
Final norm: 2d or d # before the head
LM head: V · d # d -> logits over V; often TIED to embeddingThe embedding is a lookup table mapping each of V tokens to a d-vector — it initializes the residual stream and costs essentially no FLOPs (it is indexing, not a matmul). The final normalization cleans up the grown residual stream, and the LM head projects the top-of-stack d-vector to a V-way logit distribution. Many models tie the head to the embedding (share the same V×d matrix), saving one V·d block. For small models with a big vocabulary, these two V·d terms can rival the entire block stack — a fact the worked example makes vivid.
FLOPs per token: the 2×params rule
Here is the single most valuable cost heuristic in all of transformer math: a forward pass costs about two FLOPs per parameter per token. The reason is that nearly all the compute is matrix multiplies, and each weight participates in one multiply and one add — two floating-point operations — for every token that flows through it. So:
forward FLOPs / token ≈ 2 · (matmul params)
training FLOPs / token ≈ 6 · (matmul params) # 2 fwd + 4 bwdPer block that is 2 · 12 · d^2 = 24 · d^2 FLOPs per token for the linear parts — one-third in the attention projections, two-thirds in the FFN, mirroring the parameter split. Across the stack it is L · 24 · d^2, plus 2 · V · d for the LM head (the embedding lookup is free). The 3× jump from 2 to 6 for training is the standard accounting for the backward pass computing gradients with respect to both activations and weights. When someone quotes ‘training this model took 6 · N · D FLOPs’ for N parameters and D tokens, this rule is where that comes from.
The attention term the rule leaves out
The 2×params rule counts the weight matmuls — the projections, the FFN — but it does not count the attention score and aggregation step, because those involve no weights: they are Q times K^T and the softmax-weighted sum over V. That extra compute is quadratic in sequence length: roughly 4 · N · d FLOPs per token per layer (an N×d-ish dot product for scores plus an equal one for the value aggregate, summed across heads).
Whether this term matters is entirely a question of context length. Compare it per layer: the weight matmuls cost ~24 · d^2 per token, the attention scores cost ~4 · N · d. They are equal when N ≈ 6d. Below that the FFN-plus-projection weights dominate and the 2×params rule is nearly exact; far above it — long documents, 128K-token contexts — the quadratic attention term takes over and the estimate must add it back explicitly. For a small CPU-class model with d = 512, the crossover is around N ≈ 3000 tokens: short chats are firmly weight-bound, long contexts are not.
Stacking blocks into a model
With the block defined, the full forward pass is almost anticlimactic. Embed the token ids into the residual stream [B, N, d] (adding positional information — learned, sinusoidal, or applied inside attention as RoPE). Run the stream through block 1, whose output is [B, N, d]; feed that to block 2; and so on through all L blocks. Because every block is width-preserving, the tensor’s shape never changes as it climbs the stack — only its contents get progressively refined as each sublayer adds its contribution to the stream.
After the last block, apply the final norm and then the LM head, producing logits [B, N, V] — a distribution over the vocabulary at every position. During training you compare all N positions’ predictions to the next-token targets; during autoregressive generation you take the last position’s logits, sample a token, append it, and run again (this is where the KV cache earns its keep, so the earlier positions are not recomputed). Embedding, L blocks, final norm, head — that four-part skeleton is the entire architecture.
A worked budget: a small concrete config
Take a deliberately small model — the kind you might actually run on a CPU: d = 512, L = 6 layers, h = 8 heads (d_head = 64), d_ff = 2048 (the 4× ratio), vocabulary V = 32000, tied embedding/head. Parameters:
Attention / block 4 · 512^2 = 1,048,576
FFN / block 2 · 512 · 2048 = 2,097,152
Norms / block 2 · (2 · 512) = 2,048 (LayerNorm)
Block total ≈ 3,147,776 ( ≈ 12 · d^2 )
× 6 blocks = 18,886,656 (non-embedding)
Token embedding 32000 · 512 = 16,384,000
Final norm 2 · 512 = 1,024
TOTAL (tied head) ≈ 35.3 millionTwo lessons jump out. First, the split inside the stack is exactly one-third attention (6.3M) to two-thirds FFN (12.6M). Second, the embedding is nearly as large as the entire six-layer stack — 16.4M versus 18.9M — because the vocabulary is big and the model is small. That is why tiny models report such different ‘total’ and ‘non-embedding’ counts, and why the embedding stops mattering only once L · 12 · d^2 dwarfs V · d in larger models.
The same config, in FLOPs
Now cost one token’s forward pass, at a modest context of N = 512:
Block linear 2 · 3,145,728 · 6 layers = 37,748,736
LM head 2 · 32000 · 512 = 32,768,000
Weight FLOPs (= 2 · matmul params) ≈ 70.5 MFLOP / token
Attention QK^T + ·V ~4 · N · d · L = 4 · 512 · 512 · 6 = 6,291,456
TOTAL forward ≈ 76.8 MFLOP / tokenThe weight matmuls (70.5M) are ~92% of the work; the quadratic attention term (6.3M) is a small slice at this length — but recall it grows with N, so at N = 4096 the attention term alone would be ~50M and start to rival the weights. To generate a 200-token reply you pay this per output token (plus the one-time prefill over the prompt); to train, multiply the per-token weight FLOPs by 3 (the 6×params rule) and by the number of training tokens. A single laptop CPU doing a few GFLOP/s can clearly handle ~77 MFLOP per token at interactive speeds — which is precisely why a model of this size is a realistic CPU-SLM target, and why understanding where the FLOPs go is the first step to making it fast.
Pitfalls when you compose the pieces
A few mistakes recur. Norm placement: writing Norm(x + Sublayer(x)) (post-norm) when you meant pre-norm changes training stability — the residual highway must stay un-normalized. Forgetting the final norm: pre-norm leaves the residual stream un-normalized, so a model with no final norm before the head feeds wildly-scaled vectors into the logit projection; that top norm is not optional. Double-counting the embedding: if the head is tied to the embedding, count V·d once, not twice, or your parameter number is off by a big chunk on small models.
Also: counting attention as the expensive part. At ordinary context lengths the FFN holds two-thirds of the weights and does two-thirds of the matmul FLOPs; attention’s quadratic term only dominates at long context. And concatenation vs addition: heads are concatenated inside attention, but every sublayer’s final output is added to the stream — confusing the two breaks the width-preserving invariant that makes stacking work. Get these right and the block composes cleanly, every time.