Half the transformer is attention; the other half — and usually two-thirds of the parameters — is the feed-forward network. After attention has let every token look at every other token, each token is handed to an FFN (equivalently an MLP) that processes it alone: a linear layer that expands the representation to roughly four times its width, a nonlinearity, and a second linear layer that projects back down. It is the plainest module in the architecture — two matrix multiplies and a pointwise function — yet it holds most of the model’s weights, does most of the arithmetic at ordinary context lengths, and is increasingly understood as the place where the network stores what it knows. This piece derives the FFN from first principles: its shapes and formula, why the expansion factor is there, why it runs per-position, the key-value memory intuition, the exact parameter and FLOP counts with a worked example, the gated SwiGLU family, and why all of this matters acutely when the target is a small model on a CPU.
Where the FFN sits in the block
A transformer layer is two sub-blocks stacked in sequence, each wrapped in a residual connection and a normalization. The first sub-block is multi-head self-attention, which mixes information across positions — it is how a token gathers context from its neighbours. The second sub-block is the feed-forward network, and it does the opposite: it mixes information within a position, transforming each token’s vector on its own with no reference to any other token.
Written out, one layer computes h = x + Attention(LN(x)) then y = h + FFN(LN(h)). That division of labour is the mental model to hold onto: attention decides what to combine, and the FFN decides what to do with the combined result. Because attention is a weighted average of value vectors, it is on its own a fairly linear, low-rank operation; the FFN is where the nonlinear, high-capacity feature computation lives. Stacking the two — gather, then transform, gather, then transform, layer after layer — is what gives a transformer its depth. Strip the FFN out and you are left with repeated averaging that cannot compute much of interest.
The two-layer formula and its shapes
The standard FFN is exactly a two-layer perceptron applied to one token vector x of width d_model:
FFN(x) = W_2 · φ(W_1 · x + b_1) + b_2
x : [d_model] input token vector
W_1 : [d_ff, d_model] up-projection
b_1 : [d_ff]
h = φ(W_1 x + b_1) : [d_ff] hidden activations
W_2 : [d_model, d_ff] down-projection
b_2 : [d_model]
FFN(x) : [d_model] output, same width as inputThe shape story is the whole story. The first matmul lifts the vector from d_model up to a wider hidden dimension d_ff; the nonlinearity φ acts elementwise on those d_ff numbers; the second matmul brings the vector back down to d_model so it can be added to the residual stream. Input and output widths must match because of that residual add — the FFN is a refinement of the token’s representation, not a reshaping of it. Everything interesting happens in the wide middle, and how wide that middle is (the ratio d_ff / d_model) is the single most important design knob in the module.
The expansion factor: why d_ff ≈ 4 × d_model
Since Vaswani et al.’s original transformer, the near-universal choice has been d_ff = 4 × d_model. GPT-2 small uses 768 → 3072 → 768; BERT-base is identical; GPT-3 175B uses 12288 → 49152 → 12288. Why four?
The expansion buys capacity and separability. A wider hidden layer gives the nonlinearity more dimensions to carve the input space into, so the FFN can represent more distinct features and more complex functions of the token. Projecting up before the nonlinearity, then back down, is a classic ‘expand, act, compress’ pattern: the model computes many candidate features in the high-dimensional space and then mixes the useful ones back into the residual stream. Four is not sacred — it is an empirical sweet spot. Below roughly 3× the module is measurably under-capacity for its depth; far above 4× the extra parameters buy diminishing returns while inflating the compute and memory budget. The factor also keeps the two sub-blocks roughly balanced: with d_ff = 4d, the FFN’s parameters (8 d^2) come out to twice the attention block’s projection parameters (4 d^2), which is where the ‘two-thirds of the weights live in the FFN’ figure comes from.
Why a nonlinearity is non-negotiable
Remove φ and the FFN collapses. Two stacked linear maps are just one linear map: W_2 (W_1 x) = (W_2 W_1) x = W x, a single d_model × d_model matrix. All the width in the middle would be wasted because the composite is at most rank d_model. The nonlinearity is precisely what makes the expand-then-compress structure worth having: it breaks the composition so the wide hidden layer can encode genuinely new features rather than a redundant re-parameterization.
Intuitively, each row of W_1 is a learned detector that fires (via φ) when the token matches some pattern; each column of W_2 is the response written back into the residual stream when that detector fires. Without a nonlinearity there is no notion of ‘firing’ — every detector contributes linearly and always, and the module cannot make the conditional, if-this-then-that decisions that language modelling demands. This is the same reason a plain MLP needs activations to be a universal approximator: nonlinearity between linear layers is what turns a stack of matrices into a function that can bend.
ReLU vs GELU: the choice of activation
Three activations dominate FFN history:
ReLU(x) = max(0, x)
GELU(x) ≈ 0.5 · x · (1 + tanh(√(2/π) · (x + 0.044715 x^3)))
SiLU / Swish(x) = x · σ(x), σ(x) = 1 / (1 + e^-x)ReLU is the original: cheap, sparse (it zeros every negative pre-activation), but hard — its gradient is exactly zero for negative inputs, so a unit stuck negative stops learning (the ‘dying ReLU’ problem). GELU, used in GPT-2, BERT and GPT-3, is a smooth cousin: instead of a hard gate it weights x by roughly the probability that a standard normal is below it, so small negatives leak through and the function is differentiable everywhere. That smoothness tends to train a little better at no real inference cost. SiLU/Swish is smoother still and is the building block of the gated variants discussed later. For a from-scratch small model, GELU is the safe default; ReLU is defensible when you want the cheapest possible activation and maximal hidden-layer sparsity, which can be exploited on a CPU.
Position-wise: the same MLP, applied to every token alone
The word that matters most in ‘position-wise feed-forward network’ is position-wise. The same weights W_1, W_2 are applied independently to the vector at every position in the sequence. If the input to the sub-block is a matrix X of shape [N, d_model] for a sequence of N tokens, the FFN is just X passed through the two matmuls row by row — token 5 is transformed with no knowledge of token 6.
This is deliberate and it is what makes the design coherent. Attention is the only place tokens exchange information; the FFN is a pure per-token computation. That separation is why a transformer can process a whole sequence in parallel and why the FFN is implemented as a single big batched matmul ([N, d_model] × [d_model, d_ff]) rather than a loop. Sharing one MLP across all positions is also a massive parameter saving: the model does not learn a different transformation for ‘the third word’ than for ‘the tenth’. The transformation is a general one — ‘given a token in this contextual state, compute this update’ — and it applies wherever that state occurs in the sequence.
The FFN as the model's memory: a key-value store
A productive way to read the FFN, popularized by work on ‘transformer feed-forward layers as key-value memories,’ is as an associative lookup. Recall FFN(x) = W_2 · φ(W_1 x) (dropping biases). Treat the rows of W_1 as keys and the columns of W_2 as values. The product W_1 x scores the token against every key; φ gates which keys fire; and W_2 reads out a weighted sum of the corresponding values back into the residual stream.
Under this lens the hidden dimension d_ff is the number of memory slots, and each slot is a learned (pattern → response) pair. Empirically, individual hidden units respond to interpretable patterns — a particular n-gram, a topic, a syntactic role — and their value vectors nudge the output distribution toward the tokens that tend to follow. This is the leading intuition for where facts live in a language model: not in attention, which routes information, but in the FFN, which stores it. It also explains why widening d_ff (more slots) reliably adds knowledge capacity, and why model-editing techniques that change what a model ‘knows’ tend to target FFN weights.
Parameter count: where two-thirds of the weights go
The FFN’s parameters are dominated by its two weight matrices (the biases are negligible):
params(W_1) = d_model × d_ff
params(W_2) = d_ff × d_model
params(FFN) = 2 × d_model × d_ff (+ d_ff + d_model biases)
with d_ff = 4 d_model:
params(FFN) = 2 × d × 4d = 8 d^2 per layerCompare that to the attention sub-block, whose four projections (W_Q, W_K, W_V, W_O, each d × d) total 4 d^2 parameters. So with the standard 4× expansion the FFN carries twice the parameters of attention, i.e. roughly two-thirds of every layer’s weights (8 d^2 out of 12 d^2). Across the whole model, the FFN stack is the single largest consumer of parameters — which is exactly why it is the first thing you look at when you need to shrink a model, and why the ‘memory’ intuition above is more than a metaphor: most of the storage is literally here.
FLOPs: the per-token cost of the two matmuls
A matrix-vector product of shape [m, n] × [n] costs about 2 · m · n floating-point operations (one multiply and one add per weight). Applying that to both FFN layers, per token:
up-projection : 2 · d_model · d_ff
down-projection : 2 · d_ff · d_model
FFN FLOPs/token = 4 · d_model · d_ff
with d_ff = 4 d: FFN FLOPs/token = 16 d^2Two things fall out immediately. First, the FFN cost is linear in sequence length — total FFN FLOPs for a sequence are just N × 16 d^2, since each token is processed independently. There is no N^2 term anywhere in the FFN; the quadratic cost of a transformer lives entirely in attention. Second, the FFN cost is proportional to the parameter count (2 × params FLOPs per token), which is the general rule of thumb for any dense linear layer: every weight costs about two FLOPs per token that passes through it. That single fact is the backbone of almost every transformer cost estimate.
Why the FFN dominates compute at short context
Split a layer’s per-token FLOPs into three parts. The FFN is 16 d^2. The attention projections (Q, K, V, O) are 8 d^2. The attention score-and-average step (QK^T then times V) is about 4 N d per token, because each token must interact with all N keys and values.
FFN : 16 d^2 (independent of N)
attn proj : 8 d^2 (independent of N)
attn scores : 4 N d (grows with context)
FFN > attn-scores when 16 d^2 > 4 N d => N < 4dSo the quadratic attention term only overtakes the FFN once the context N approaches a few times d_model. For most real configurations that crossover is in the low thousands of tokens; below it, the FFN is the largest single line in the FLOP budget. This is why, at typical prompt lengths, a transformer is essentially a stack of big FFN matmuls with attention as a comparatively cheap router — and why optimizing the FFN (kernels, quantization, sparsity) is where most inference speed-ups on short-to-medium context are found. Attention’s N^2 only becomes the headline cost at long context.
A worked param / FLOP example
Take GPT-2-small dimensions: d_model = 768, d_ff = 3072 (exactly 4×), 12 layers.
PARAMS (one FFN):
2 × 768 × 3072 = 4,718,592 ≈ 4.72M (+ ~3.8K biases)
attention projections: 4 × 768^2 = 2,359,296 ≈ 2.36M
ratio FFN : attn = 2 : 1
FFN params across 12 layers: 12 × 4.72M ≈ 56.6M
(of ~124M total params — the FFN stack is the biggest block)
FLOPS per token (one FFN):
4 × 768 × 3072 = 9,437,184 ≈ 9.44 MFLOPs
crossover: attn-scores overtake FFN when N > 4 × 768 = 3072 tokens
so for any prompt shorter than ~3K tokens, the FFN out-computes
the quadratic part of attention.Scale the same arithmetic to GPT-3 175B (d = 12288, d_ff = 49152) and one FFN is 2 × 12288 × 49152 ≈ 1.21B parameters — a single FFN layer larger than an entire small model. The structure never changes; only d grows, and because the cost scales as d^2, the FFN’s share of the budget only gets more dominant.
Gated siblings: GLU and SwiGLU
The modern refinement replaces the single up-projection with a gated linear unit (GLU). Instead of one linear layer feeding the activation, there are two parallel projections — a gate and an up — multiplied elementwise:
standard : FFN(x) = W_2 · φ(W_1 x)
GLU : FFN(x) = W_2 · ( σ(W_gate x) ⊙ (W_up x) )
SwiGLU : FFN(x) = W_2 · ( Swish(W_gate x) ⊙ (W_up x) )
⊙ = elementwise product; Swish(z) = z · σ(z)SwiGLU — a GLU using the Swish/SiLU gate — is the current default in Llama, Mistral, PaLM, Phi and most recent open models. The gate learns a data-dependent, multiplicative mask over the up-projected features: the network can turn features on or off conditioned on the input, a strictly richer operation than a fixed pointwise nonlinearity. Empirically this buys roughly a 1% perplexity improvement at matched compute — small but consistent, and effectively free at inference. It is the same expand-act-compress skeleton as the vanilla FFN, with a smarter, multiplicative activation in the middle; think of it as a sibling of the standard MLP, not a different animal.
The 8/3 rule: keeping SwiGLU's budget honest
SwiGLU has three weight matrices — W_gate, W_up and W_2 — instead of two. At a naive d_ff = 4d that would be 3 × 4 d^2 = 12 d^2 parameters, a 50% increase over the standard FFN’s 8 d^2. To compare architectures fairly you hold the parameter (and FLOP) budget fixed, which means shrinking the hidden width.
standard params : 2 × d × d_ff
SwiGLU params : 3 × d × d_ff'
match: 3 d · d_ff' = 2 d · (4d) => d_ff' = (8/3) d ≈ 2.67 dThat is why Llama-style models quote a feed-forward multiplier of about 2.67× rather than 4×, often rounded to a hardware-friendly multiple. The lesson is a general one for reading model cards: always ask how many projections the FFN has before comparing hidden dimensions. A SwiGLU block at 2.67× and a vanilla block at 4× cost the same; the SwiGLU simply spends its budget on a gate instead of a wider slab, and tends to come out slightly ahead in quality for it.
CPU and small-model implications
For a small language model on a CPU, the FFN is where the pain and the opportunity both concentrate. Because it is the majority of the weights, it is the majority of the bytes you must stream from memory on every token — and CPU inference at batch size 1 is almost entirely memory-bandwidth-bound, not compute-bound. Each generated token re-reads the FFN weight matrices, so decode speed tracks how fast you can move those matrices, which makes the FFN the prime target for quantization: dropping W_1 and W_2 from FP32 to INT8 or INT4 cuts the bytes moved (and the cache footprint) by 4–8×, often the single biggest lever for tokens-per-second on a laptop.
Two more CPU-specific angles. First, the FFN matmuls are large, regular, and cache-friendly — they map well onto SIMD and threaded BLAS, so a good kernel matters more here than anywhere else. Second, activation sparsity is exploitable: with ReLU (and, softly, with gated units) many hidden units are zero for a given token, and skipping the zero columns of W_2 can save real work on a CPU that lacks a GPU’s brute-force throughput. When you profile a small model on a CPU and ask ‘where did the time go,’ at ordinary context lengths the honest answer is usually: the feed-forward layers.
Common pitfalls and misconceptions
A few traps recur. ‘The FFN is just a formality after attention.’ The opposite is true — it holds most of the parameters and does most of the arithmetic at normal context lengths; it is where the model computes and stores. ‘Bigger d_ff is always better.’ Capacity helps until it does not; past ~4× the returns fade while compute, memory and overfitting risk keep climbing, which is why the ratio is a tuned constant, not ‘as large as possible.’
‘The nonlinearity is a minor detail.’ Remove it and the entire wide layer collapses to a single small matrix — it is load-bearing, not decorative. ‘SwiGLU has more parameters, so it is unfair to compare.’ Only if you forget the 8/3 rescaling; matched for budget it is a fair, and usually favourable, trade. Finally, ‘attention is the expensive part.’ It is — but only once context is long; for short and medium prompts the FFN out-computes attention’s quadratic term, and any performance work that ignores the FFN is optimizing the wrong half of the block.