Why architecture matters here
Architecture matters here because the FFN is not a minor component — it typically holds roughly two-thirds of a transformer block's parameters and a comparable share of its FLOPs, so any change to it moves the whole model's quality-per-parameter and quality-per-FLOP curves. When you are training a model at the scale where a single run costs millions of dollars, a change that buys a consistent perplexity improvement at identical cost is enormously valuable: it is free quality. SwiGLU is precisely such a change, which is why it propagated across essentially every frontier open model despite adding conceptual complexity.
The deeper reason to care is what the gate represents. A plain FFN computes W2 · act(W1 · x) — a fixed nonlinearity applied pointwise. The gated form computes W2 · (act(W · x) ⊙ (V · x)), introducing a multiplicative interaction between two learned projections of the same input. Multiplicative gating lets the network implement conditional, input-dependent routing of information at the level of individual hidden dimensions, which additive-only architectures approximate only with more depth or width. That extra expressive power is what pays for the third matrix.
Finally, the block's cost structure is what makes the two-thirds rescaling non-negotiable. If you added the third matrix without shrinking d_ff, you would be comparing a bigger model to a smaller one and any gain would be confounded with the extra capacity. By holding parameters and FLOPs fixed, the architecture makes an honest claim: at the same budget, gating beats a single activation. Understanding this budget discipline is essential to reasoning about model sizing, because it changes the canonical d_ff = 4·d_model rule of thumb into roughly d_ff ≈ (8/3)·d_model for SwiGLU blocks.
This budget reasoning also reframes how you think about the attention-versus-FFN split within a block. Because the FFN carries the majority of parameters, decisions about its width ripple through the entire parameter and memory budget of the model far more than tweaks to attention head counts do. When practitioners report that SwiGLU 'just works better,' the honest interpretation is that, at a fixed budget, spending some of the FFN's capacity on a multiplicative gate is a better allocation than spending all of it on width behind a single activation. That is a statement about how to distribute a fixed resource, and it is why the two-thirds rescaling is not a footnote but the core of the claim — without it, the comparison is meaningless.
The architecture: every piece explained
Begin with the SiLU (Swish) activation: SiLU(z) = z · sigmoid(z). Unlike ReLU it is smooth everywhere and non-monotonic — it dips slightly negative for small negative inputs before rising — which gives gradients a gentle, well-behaved landscape and avoids ReLU's dead-neuron problem where a unit stuck in the negative region receives zero gradient forever. SiLU approaches the identity for large positive z and zero for large negative z, so it behaves like a soft, differentiable gate on its own input.
The gated block takes the input x (shape [tokens, d_model]) and applies two independent linear up-projections to width d_ff: the gate projection W, whose output is passed through SiLU, and the value projection V, left linear. The two are combined by an elementwise (Hadamard) product: h = SiLU(x·W) ⊙ (x·V). Each of the d_ff hidden dimensions is thus the product of a SiLU-activated feature and a linear feature — the SiLU branch acts as a soft, learned gate that scales the value branch dimension by dimension. A third matrix, the down projection W2, maps h back from d_ff to d_model, and the result is added to the residual stream.
Parameter budgeting ties it together. A classic FFN has two matrices of size d_model×d_ff and d_ff×d_model, totaling 2·d_model·d_ff parameters, conventionally with d_ff = 4·d_model. SwiGLU has three matrices — W and V are each d_model×d_ff, W2 is d_ff×d_model — totaling 3·d_model·d_ff. To match the classic count you set the SwiGLU hidden width to two-thirds of what you would otherwise use, i.e. d_ff ≈ (2/3)·(4·d_model) = (8/3)·d_model, so 3·d_model·(8/3·d_model) = 8·d_model² equals the classic 2·d_model·(4·d_model). Modern implementations also drop the bias terms (they add little and complicate fusion) and often round d_ff to a hardware-friendly multiple. Note that the two up-projections W and V can be computed as a single fused matmul that is then split, which is how efficient kernels avoid launching two separate GEMMs.
End-to-end flow
Walk one token's hidden vector through a SwiGLU block. It enters as a residual-stream vector of width d_model, first passing through the block's pre-normalization (RMSNorm in most modern models). The normalized vector is multiplied by the fused up-projection matrix, producing a 2·d_ff-wide intermediate that is split into the gate pre-activation (x·W) and the value (x·V), each of width d_ff. The gate half goes through SiLU; the value half is left as-is.
The two d_ff-wide vectors are multiplied elementwise: dimension i of the output is SiLU((x·W)_i) · (x·V)_i. Intuitively, the SiLU branch decides how much of each value-branch feature to let through — near zero for dimensions the gate suppresses, near the value itself for dimensions it passes — a per-dimension, input-conditioned modulation. This gated d_ff-wide vector is then projected by W2 back to d_model.
Finally the down-projected d_model vector is added to the residual stream that entered the block, so the FFN's contribution is a learned increment to the running representation rather than a replacement. In the backward pass, gradients flow through both up-projections — importantly, the value branch V receives gradient scaled by the gate value, and the gate branch W receives gradient scaled by the value and by SiLU's derivative — so the two projections co-adapt: W learns what to gate, V learns what to gate. Because the elementwise product couples them, the optimizer shapes a joint gate/value representation rather than two independent features, which is the mechanism behind the observed quality gain.
It helps to contrast the information flow with a plain FFN concretely. In W2·GELU(W1·x), each hidden unit is a fixed nonlinearity of a single linear combination of the input; the only way for the network to make one feature conditional on another is to stack more layers. In SwiGLU, the gate branch and the value branch are two different linear reads of the same input, and their elementwise product creates a second-order (multiplicative) interaction within a single layer — the output can depend on the product of two input features, not just their weighted sum. This is a strictly richer per-layer function class, and it is available at the same FLOP budget, which is the compact explanation for why gated FFNs consistently edge out ungated ones across scales.