Residual Connections and Gradient Flow

The skip connection: y = x + F(x)

A residual connection (or skip connection) wraps a sublayer F so that its output is added to its own input:

y = x + F(x)

x : the input vector       (shape [d])
F : the sublayer           (attention, or the FFN)
y : the block output       (shape [d], same as x)

The critical constraint is that F(x) must have the same shape as x, because you cannot add a [d] vector to anything else. This is why the model dimension d_model is held constant all the way down the stack: every block takes a [N, d] tensor and returns a [N, d] tensor, so the add always lines up.

Conceptually, F(x) is not the answer — it is a proposed edit to the answer so far. The block reads the current state x, computes how it would like to change it, and commits x + Δ where Δ = F(x). If a block has nothing useful to add, the cheapest thing it can learn is F(x) ≈ 0, and the signal passes through untouched. That ‘do no harm’ default is the seed of everything else in this article.

Where residuals live in a transformer block

A single transformer block contains two residual connections, one around each sublayer. In the now-standard pre-norm arrangement:

x = x + Attention(Norm(x))   # residual #1, around attention
x = x + FFN(Norm(x))         # residual #2, around the MLP

Read those two lines carefully: the thing being added back is the un-normalized x, while the sublayer sees a normalized copy. The normalization is tucked inside the residual branch; the skip path itself is a clean, unmodified copy of x. That clean path is the whole point — keep it in mind when we reach the pre-norm-versus-post-norm discussion.

A model with L layers therefore has 2L residual adds threaded through it. A 32-layer small model has 64 of these skip points, each one a place where the forward signal can flow straight through and, crucially, where the backward gradient can too. The residual is not a decoration bolted onto the interesting part of the block; it is the spine the interesting parts hang off.

The local Jacobian: differentiating the add

Training is gradient descent, so what matters is how a small change in x moves y — the derivative (in vector terms, the Jacobian). Differentiate y = x + F(x) with respect to x:

y   = x + F(x)

dy      d           dF
---  =  --(x)   +   --   =   I  +  F'(x)
dx      dx          dx

The derivative of x with respect to itself is the identity matrix I (a plain 1 in the scalar case). The derivative of F is its own Jacobian F’(x). So the local Jacobian of a residual block is I + F’(x) — the sublayer’s Jacobian with the identity added on.

Compare a non-residual block, y = F(x), whose Jacobian is just F’(x). The residual has bought us a guaranteed +I term that is present no matter what F does. That single additive identity is what the entire ‘gradient highway’ story rests on, and the next section chases it through the backward pass.

The gradient highway: dL/dx = dL/dy (1 + F')

Backpropagation multiplies the upstream gradient by the local Jacobian. If L is the loss and the gradient arriving at the block output is ∂L/∂y, then the gradient handed to the block input is:

dL       dL     dy      dL
--   =   --  ·  --   =  --  ·  ( I + F'(x) )
dx       dy     dx      dy

       =   dL/dy   +   dL/dy · F'(x)
           \_____/     \____________/
          direct path    through-F path

The gradient splits into two summands. The second term, ∂L/∂y · F’(x), is the ordinary path through the sublayer — and it can shrink toward zero if F’ has small singular values, exactly the vanishing-gradient failure mode. But the first term is ∂L/∂y unmodified. It is a direct wire from the output’s gradient to the input’s gradient, with no matrix in the way to attenuate it.

That is the highway. Even in the worst case where the sublayer learns nothing and F’(x) = 0, the block still passes ∂L/∂y straight back: ∂L/∂x = ∂L/∂y. The gradient can never fully vanish at a residual block, because the +1 guarantees a lower bound on how much signal survives the hop.

A worked numeric example: watch the +1 do its job

Make it concrete with scalars. Take a single unit, input x = 2.0, and suppose the sublayer is F(x) = 0.1·x so that F’(x) = 0.1 — a deliberately weak sublayer, the kind whose gradient normally fades. Say the gradient arriving from above is ∂L/∂y = 1.0.

Without residual   y = F(x):
  dL/dx = dL/dy · F'(x)      = 1.0 · 0.1        = 0.10

With residual      y = x + F(x):
  dL/dx = dL/dy · (1 + F'(x)) = 1.0 · (1 + 0.1)  = 1.10

One weak layer barely differs (0.10 vs 1.10), but the effect compounds through depth. Stack ten identical weak layers. Without residuals the gradient is multiplied by 0.1 ten times; with residuals, by 1.1 ten times:

no residual:   0.1^10  ≈ 1e-10   (gradient annihilated)
residual:      1.1^10  ≈ 2.59    (gradient fully alive)

That is a ten-orders-of-magnitude difference over just ten layers. The 1e-10 gradient means the earliest layer effectively never learns; the 2.59 gradient means it trains just fine. The +1 turned a vanishing product into a healthy one.

What vanishing gradients actually are

The vanishing-gradient problem is a property of deep function composition. When you stack layers x_L = f_L(f_{L-1}(...f_1(x_0))), the chain rule makes the gradient at the bottom a product of every layer’s Jacobian:

dL/dx_0 = dL/dx_L · f_L' · f_{L-1}' · ... · f_1'

A product of many factors is unstable. If the typical factor has magnitude below 1, the product decays geometrically toward zero (vanishing); if above 1, it explodes. Either way the early layers get a gradient that is wrong by orders of magnitude, and training stalls or diverges. This is precisely why, before residuals and good normalization, networks beyond a modest depth were notoriously hard to train.

Residuals change each factor from f_k’ into I + F_k’. The +I keeps every factor anchored near 1 instead of near 0, so the product no longer collapses. It does not forbid the through-F terms from shrinking — it just guarantees they are shrinking on top of a solid identity floor rather than multiplying an already-tiny number. The failure mode is defused at its source: the product of Jacobians.

Stacking depth: expanding the product of Jacobians

Do the residual version of that product explicitly. With x_k = x_{k-1} + F_k(x_{k-1}), each factor is (I + F_k’), and the end-to-end gradient is their product:

dL/dx_0 = dL/dx_L · (I + F_L') · (I + F_{L-1}') · ... · (I + F_1')

Multiply out that product and you get a sum of many terms. One of them is the product of all the I’s — which is just I. So the expansion contains a bare dL/dx_L · I = dL/dx_L term: the output gradient reaches the very first layer directly, unfiltered by any sublayer Jacobian at all. Every other term picks up one or more F_k’ factors and may be small, but that pure-identity term is always there and never shrinks.

This is the depth-scaling payoff. In a plain deep net the only route to layer 1 runs through all L Jacobians; in a residual net there are exponentially many routes of every length, including the length-zero identity route. Gradients reach early layers regardless of depth, which is exactly why residual architectures made 50-, 100-, and 1000-layer networks trainable when plain stacks of that depth would not converge.

Identity initialization: starting as a no-op

There is an elegant initialization intuition hiding in y = x + F(x). Suppose you initialize a block so that F(x) ≈ 0 at the start of training — for instance by initializing the sublayer’s final projection to (near) zero. Then the block begins life as y ≈ x: a pure identity map that copies its input to its output.

Why is that good? Because the identity function is a safe, well-conditioned starting point. A freshly initialized deep stack of identity blocks is just a long wire: the input reaches the output unchanged, the loss is sensible, and — from the Jacobian I + F’ = I + 0 = I — the gradient reaches every layer at full strength on the very first step. Training then gently perturbs each block away from identity only as far as it helps.

Contrast a non-residual stack, where a block must learn to approximate the identity from scratch if that is what the task needs — a surprisingly hard thing for a random-initialized nonlinear layer to do. Residuals make identity the default and any deviation an earned improvement. This ‘start as a no-op, add signal only when it pays’ behavior is a large part of why residual nets train so stably.

The residual stream: a shared read/write bus

Zoom out from one block and a different picture appears. Because every block adds to x rather than overwriting it, the vector flowing down the stack is a running accumulation:

x_0                        (token embedding)
x_1 = x_0 + F_1(x_0)
x_2 = x_1 + F_2(x_1)  =  x_0 + F_1(...) + F_2(...)
 ...
x_L = x_0 + Σ_k F_k(x_{k-1})     (embedding + every block's edit)

This x is the residual stream. Interpretability research treats it as a shared communication channel that runs the full height of the model: each sublayer reads from the stream (through its normalized input), computes something, and writes its result back by addition. Nothing is ever destroyed; contributions simply sum into the same d-dimensional space.

The view is powerful. An early attention head can write a feature into the stream that a much later FFN reads and uses — the two communicate across many layers through the linear, additive stream. Because writes are additive and the space is shared, different components can use different subspaces of the same d dimensions as private channels. The plus sign in y = x + F(x) is what makes the stream a durable bus rather than a lossy relay.

Pre-norm vs post-norm: where the LayerNorm goes

Normalization and the residual can be wired in two orders, and the choice materially affects trainability.

Post-norm (original Transformer):
  x = Norm( x + F(x) )        # normalize AFTER the add

Pre-norm (modern default):
  x = x + F( Norm(x) )        # normalize INSIDE the branch

In post-norm, the normalization sits on the residual path: the sum x + F(x) is immediately passed through Norm. That means the clean identity route is interrupted at every layer by a normalization whose own Jacobian scales and rotates the gradient. The pristine +I highway is no longer pristine — it is repeatedly filtered.

In pre-norm, the normalization is moved inside the branch, applied to x before F sees it. The addition x + F(Norm(x)) leaves the skip path itself completely unnormalized, so the identity route from output to input stays clean all the way down. The Jacobian of the skip path is exactly I, with the norm’s Jacobian confined to the branch. This is the arrangement the two lines in the article’s block example use.

Why pre-norm won for deep stacks

The consequence of that placement is a real difference in how deep you can go without heroics. Because pre-norm keeps an undisturbed identity path, the gradient highway survives the full depth of the network, and pre-norm transformers can be trained to great depth with standard optimizers and, often, without a learning-rate warmup. It is the default in essentially every modern large and small language model for exactly this robustness.

Post-norm is not useless — the original Transformer used it, and it can yield slightly better final quality at moderate depth because the repeated normalization keeps activations tightly controlled. But it is fragile to train deep: without careful warmup, small initialization, or tricks, the interrupted highway lets gradients misbehave and training can diverge early. The engineering trade is stability-and-scale (pre-norm) versus a touch of final-quality under careful tuning (post-norm).

One caveat pre-norm introduces: since every block adds into the stream and the skip path never re-normalizes, the stream’s magnitude tends to grow with depth. Models handle this with a final Norm before the output head and, sometimes, small residual scaling — a manageable cost for the trainability the clean highway buys.

Refinement, not replacement: what the stream means for representations

The additive structure reframes what a deep network is. A plain stack computes a sequence of total transformations, each layer discarding its predecessor and building a fresh representation. A residual stack instead computes a base representation (the embedding) and then a long series of small refinements to it. Depth becomes iterative improvement rather than repeated reinvention.

This matches what we observe empirically. In residual transformers, the hidden state often changes gradually from layer to layer — consecutive residual-stream vectors are highly similar, because each block only nudges the stream by F(x). Individual blocks can frequently be removed or reordered with surprisingly little damage, a robustness that makes sense only if each block is an incremental edit rather than a load-bearing total rewrite. The network behaves more like an ensemble of many shallow paths than one rigid deep chain.

That mental model — a shared stream being progressively edited — is the right one to carry into debugging and interpretability. When you ask ‘what does layer 12 do?’ the honest answer is usually ‘it adds a specific, often-small correction to a stream that is already mostly correct.’

Shapes, cost, and common pitfalls

Mechanically the residual is almost free. It is an element-wise add of two [N, d] tensors — N·d additions, no matmul, negligible FLOPs next to attention’s O(N^2 d) or the FFN’s O(N d^2). Its cost is memory traffic, not compute: the input x must be kept live so it can be added at the end of the branch, which is one reason activation memory in a transformer is dominated by these residual tensors.

The pitfalls are about preserving the clean path. Shape must match: if a block ever changes d, the skip needs a projection (y = W_s x + F(x)), and a badly initialized W_s can reintroduce the very attenuation the identity was there to avoid. Do not normalize the skip path if you want pre-norm’s benefits — put the norm in the branch. Watch the stream norm: because writes accumulate, activations grow with depth, and in low precision (fp16/bf16) an unchecked residual stream can overflow or lose resolution, so a final norm and sane scaling matter. And remember that residuals defuse vanishing gradients, not exploding ones — the +1 raises the floor but does not cap the ceiling, so normalization and clipping still earn their keep.

Residuals and CPU-hosted small language models

For small language models running on a CPU, residual connections matter in three concrete ways. First, they are what let an SLM be deep enough to be good while staying narrow enough to be cheap. On a CPU your compute and memory bandwidth are tight, so a natural design is a slim model — small d_model, more layers — and depth is only trainable because the residual highway keeps gradients flowing. Without residuals, a deep-and-narrow SLM simply would not converge; with them, depth is a cheap axis to spend on quality.

Second, the additive stream is friendly to inference-time surgery. Because blocks are incremental edits, layer pruning, early-exit (stop once the stream stops changing much), and depth-adaptive computation are all viable ways to trim CPU latency, and they work precisely because removing one incremental edit rarely breaks the whole computation. The refinement view is not just theory here; it is a lever for making the model faster on modest hardware.

Third, the residual add is numerically forgiving under quantization. Quantizing weights to int8/int4 injects small per-block errors, but each error is just a slightly-off Δ added to a stream that is mostly carried by the identity path — so errors tend to stay bounded rather than compounding multiplicatively down a fragile chain. The one thing to respect is the growing stream magnitude: keep a final normalization and pick a precision that holds the accumulated range, and the residual structure repays you with a deep, robust, CPU-friendly model.