RMSNorm — The Modern Default

From LayerNorm to RMSNorm: the one-line change

Start from what LayerNorm does. For an activation vector x of dimension d (one token’s hidden state), LayerNorm computes a mean μ = (1/d) Σ_i x_i, a variance σ² = (1/d) Σ_i (x_i − μ)², normalizes to (x_i − μ) / √(σ² + ε), then applies a learned scale and shift γ_i · x̂_i + β_i. Two statistics, two parameter vectors.

RMSNorm keeps only the magnitude normalization. It drops the mean subtraction and, with it, the variance — because once you are not centering, the natural measure of scale is not the variance around the mean but the root-mean-square around zero. The definition is simply RMS(x) = √((1/d) Σ_i x_i² + ε) and the output is y_i = γ_i · x_i / RMS(x). That is the entire change: delete μ, delete β, and replace variance with raw mean-of-squares. Everything else about why normalization helps — keeping activations on a stable scale so gradients neither explode nor vanish as they flow through a deep residual stack — is preserved. The rest of this article is an accounting of what that one-line deletion buys you and why it costs nothing in quality.

The forward pass, precisely

Let x ∈ ℝ^d be one token’s hidden vector; in a batch the same operation runs independently over the last axis of X: [B, N, d]. The forward pass is three steps:

ms   = (1/d) * Σ_j x_j²          # mean square, a scalar per token
r    = √(ms + ε)              # the RMS (+ ε for stability)
y_i  = γ_i * (x_i / r)            # normalize, then apply learned gain

The learnable parameter is the gain vector γ ∈ ℝ^d, usually initialized to all ones so the layer starts as a pure normalizer. There is no bias β. The epsilon (10⁻⁵ to 10⁻⁸ is typical) sits inside the square root so that a vector of all zeros produces a finite, well-defined output rather than a divide-by-zero. Note the shape story: ms and r are scalars per token (a reduction over the d axis), broadcast back across all d components. Because the reduction is over the feature dimension only, RMSNorm — like LayerNorm — is completely independent of batch and sequence position, which is exactly why it behaves identically at training time and at inference with a batch of one, and why it needs no running statistics the way BatchNorm does.

Why dropping re-centering is fine

The intuition rests on an invariance argument. LayerNorm gives a network two kinds of robustness: re-centering invariance (add a constant to every component of the input and the output is unchanged) and re-scaling invariance (multiply the input by a constant and the output is unchanged). Zhang & Sennrich’s claim, backed by ablations across machine translation, language modeling, and other tasks, is that the re-scaling invariance is what actually stabilizes optimization — it keeps the effective gradient scale bounded regardless of how large activations grow — while the re-centering invariance is close to inert.

Why would centering not matter? In a transformer, the hidden vector immediately feeds linear projections whose weights can absorb any consistent mean, and the residual stream plus the learned gain give the model ample freedom to place activations where it wants. Empirically the mean of a hidden vector is small and carries little task signal; the magnitude is what must be controlled. RMSNorm keeps the re-scaling invariance exactly — scale x by any c > 0 and RMS(cx) = c·RMS(x), so y is unchanged — while deliberately giving up re-centering invariance it never needed. The result is the same training stability with strictly less machinery.

A worked numeric example (forward)

Take d = 4, x = [2, −2, 4, 0], γ = [1, 1, 1, 1], and a negligible ε. Sum of squares is 4 + 4 + 16 + 0 = 24; the mean square is 24 / 4 = 6; so r = √6 ≈ 2.449. The output is y = x / r = [0.816, −0.816, 1.633, 0]. Notice RMSNorm simply rescaled the vector — it kept the same direction and the same sign pattern, pulling its overall magnitude to a unit root-mean-square.

x        = [ 2.000, -2.000,  4.000,  0.000]
Σ x²      = 24     →  ms = 6     →  r = √6 ≈ 2.449
y = x/r  = [ 0.816, -0.816,  1.633,  0.000]   (RMS of y = 1)

Now run the same input through LayerNorm. The mean is (2 − 2 + 4 + 0)/4 = 1, so the centered vector is [1, −3, 3, −1]; its variance is (1 + 9 + 9 + 1)/4 = 5, giving √5 ≈ 2.236 and an output of [0.447, −1.342, 1.342, −0.447]. The two results differ precisely because this vector had a non-zero mean of 1: LayerNorm shifted it to zero mean first, RMSNorm did not. When the input already has zero mean, the two coincide.

LayerNorm vs RMSNorm, side by side

The contrast is easiest to read as a table. Everything RMSNorm removes is on the left; everything it keeps is on the right.

The mental model: LayerNorm is affine + full standardization; RMSNorm is magnitude standardization + a gain. The two produce the same output whenever the incoming vector is already centered, and they diverge in proportion to how far the vector’s mean sits from zero. In practice, across a trained transformer, that gap is small enough that swapping one for the other leaves the loss curve essentially untouched — which is the whole reason the swap is worth making.

Parameter savings: no bias, no shift

RMSNorm keeps one learnable vector, γ ∈ ℝ^d, and drops the bias β ∈ ℝ^d. So each normalization layer holds d parameters instead of 2d — exactly half. This is small next to the attention and MLP weight matrices (which scale as d²), but it is not nothing, and it compounds: a modern decoder puts a norm before attention and a norm before the MLP in every block, plus one final norm.

Concretely, take a Llama-style model with d = 4096 and 32 layers. That is 2 × 32 + 1 = 65 normalization layers. LayerNorm would carry 65 × 2 × 4096 ≈ 533K norm parameters; RMSNorm carries 65 × 4096 ≈ 266K — a saving of a quarter of a million parameters. Against a 7-billion-parameter model that is a rounding error in count, so the parameter reduction is not the headline benefit. The point is subtler: removing β removes a whole broadcast-add over the hidden state on every token, removes those weights from the optimizer state (which in Adam is 2× the parameter memory again), and removes one more thing the training run can get wrong. The real wins are in compute and simplicity, which the next sections quantify.

Compute savings: one reduction, not two

Normalization is a memory-bandwidth-bound, elementwise-plus-reduction operation, so the honest way to compare cost is to count passes over the d-vector and the arithmetic per element. LayerNorm needs two dependent reductions: first a pass to compute the mean μ, then — because variance depends on μ — a second pass to accumulate (x_i − μ)². Then it subtracts, rescales, scales by γ, and adds β.

RMSNorm needs a single reduction — the sum of squares Σ x_i² — because mean-of-squares does not depend on any prior statistic. There is no subtraction of μ and no addition of β. Per element the work drops from roughly compute mean, subtract, square, normalize, scale, shift to square, normalize, scale. The saved reduction is the expensive part: a reduction has a serial dependency and poor arithmetic intensity, so halving the number of reduction passes is worth more than the raw FLOP count suggests. In practice fused RMSNorm kernels report on the order of a 10–15% speedup over LayerNorm at the same width, and the fused kernel is simpler to write because there is one accumulator and no two-stage mean-then-variance dance.

The backward pass, derived

The gradient is where RMSNorm’s simplicity really shows. Write ms = (1/d)Σ_j x_j², r = √(ms + ε), x̂_i = x_i / r, and y_i = γ_i x̂_i. Given upstream gradient dy_i = ∂L/∂y_i, the parameter gradient is immediate: ∂L/∂γ_i = dy_i · x̂_i = dy_i x_i / r (summed over the batch and sequence).

For the input gradient, let g_i = γ_i dy_i. We need ∂r/∂x_i = x_i / (d·r), since ∂ms/∂x_i = 2x_i/d. Applying the quotient rule to x_i/r and summing the contributions gives:

g_i      = γ_i * dy_i
∂L/∂x_i = (1/r) * ( g_i  -  (x_i / (d * r²)) * Σ_j x_j g_j )

Read the structure: the first term g_i / r is the “direct” gradient, and the second is a single correction proportional to the projection of g onto x (the dot product Σ_j x_j g_j). There is exactly one reduction to backprop through — that dot product — because there is only one forward statistic. LayerNorm’s backward pass carries two such correction terms, one for the mean and one for the variance, so its gradient is strictly busier.

Reading and sanity-checking the gradient

A couple of checks make the formula trustworthy. First, dimensional/scaling consistency: RMSNorm is invariant to scaling x → cx, so its input gradient must be homogeneous of degree −1 in x. In the formula, r scales like c and the bracket is degree-zero in x (the correction term has x_i in the numerator and r², which is degree two, in the denominator, times a degree-one dot product), so ∂L/∂x_i scales like 1/c — exactly right.

Second, the correction term is what makes the gradient orthogonalize: it removes the component of the incoming gradient that merely tries to change the vector’s overall length, because that direction is precisely the one RMSNorm has normalized away and to which the output is therefore insensitive. If you feed a gradient g ∝ x, the bracket collapses toward zero — the layer correctly reports that pushing along x’s own direction does nothing after normalization. One caveat when implementing: accumulate Σ x_j g_j and Σ x_j² in fp32 even for a half-precision model, because a sum of squares over a wide vector is exactly the kind of thing that overflows or loses precision in fp16.

Numerical stability and where epsilon goes

The single most common implementation subtlety is the placement of ε. The canonical form — and the one used by Llama and most reference kernels — puts it inside the square root: r = √(mean(x²) + ε), equivalently multiply by rsqrt(mean(x²) + ε). This guarantees a finite output even for the all-zeros vector and bounds the gain when a token’s activations are tiny. Placing ε outside the root, as √(mean(x²)) + ε, changes the behavior near zero and is not equivalent; mixing conventions is a classic source of a model that trains fine but fails to reproduce a reference implementation.

Two more stability notes. Do the reduction in higher precision than the storage dtype, as noted above — the mean-of-squares is the fragile quantity. And remember the reduction is strictly over the last (feature) axis: normalizing across the batch or sequence axis by mistake silently couples tokens together and breaks causality at inference. Because RMSNorm holds no running statistics and no bias, there are fewer moving parts to misconfigure than with LayerNorm, but the ε-inside-the-root and fp32-reduction rules are the two that actually bite in practice.

Why modern models adopt it

The adoption story is broad and consistent. T5 (2020) was an early mover, using a simplified LayerNorm with no bias and no mean subtraction — which is RMSNorm in all but name. From there it became the decoder-LLM default: the entire Llama family (1, 2, and 3), Mistral, Qwen, Gemma, and Baichuan all use RMSNorm, and DeepMind’s Gopher/Chinchilla and Google’s PaLM adopted it as well (PaLM pairing it with a general removal of biases throughout the network).

The reasons line up with everything above. The quality is a wash — ablations repeatedly show matching or marginally better perplexity — so there is no accuracy cost to pay. Against that, you get a cheaper, easier-to-fuse kernel, half the norm parameters and optimizer state, and one fewer thing to tune. When you are training a model that will run a normalization roughly 2L times per token for hundreds of billions of tokens, a 10–15% cut on that operation and a simpler kernel are free money. RMSNorm won not because it is dramatically better but because it is strictly simpler at no measurable cost — the kind of trade that, once demonstrated, becomes the default everywhere.

Pre-norm placement in the residual stream

RMSNorm is almost always used in the pre-norm configuration: the norm sits on the input to each sublayer, inside the residual branch, as x + Sublayer(RMSNorm(x)), rather than after the addition. This keeps a clean, un-normalized residual highway running the full depth of the model, which is what makes very deep transformers trainable without warmup gymnastics; the norm conditions only what feeds attention or the MLP, not the skip path itself.

This placement interacts nicely with RMSNorm’s scale-only nature. Because the residual stream can carry whatever mean it likes and only the branch input is magnitude-normalized, the missing re-centering is even less consequential than it would be in a post-norm design: the model has an explicit, unnormalized path for any additive/mean information to travel, so asking the norm to also center would be redundant. A single final RMSNorm before the output projection (the language-model head) tidies the last hidden state. The upshot is that RMSNorm and the modern pre-norm residual architecture fit together — each makes the other’s simplifications safe.

CPU-SLM cost: why it matters more off the GPU

On a CPU-side small language model the case for RMSNorm is, if anything, stronger. CPUs have far less memory bandwidth than accelerators and no army of tensor cores to hide a clumsy kernel behind, so a bandwidth-bound elementwise op like normalization is a real, visible slice of per-token latency rather than noise. Cutting one of the two reduction passes, deleting the mean subtraction, and dropping the β load means fewer sweeps over the hidden vector and fewer bytes touched per token — exactly the quantities a CPU is starved on.

The simplicity also vectorizes better. A one-accumulator sum-of-squares maps cleanly onto a SIMD horizontal reduction (AVX2/AVX-512 or NEON), where LayerNorm’s dependent mean-then-variance pattern is fiddlier to fuse and keep in registers. With no running statistics and no bias, the whole layer is a couple of vectorized loops and a rsqrt. For a small model running many layers per token on a laptop or phone CPU, those savings show up directly in tokens-per-second, and the halved norm-parameter footprint trims the model’s memory and load time. RMSNorm is a small optimization that a bandwidth-limited CPU feels more acutely than a GPU does.

Common pitfalls

A short field guide to the mistakes that actually happen. Epsilon placement: inside the root, not outside — the two are not equivalent and mismatches break reference reproduction. Reduction precision: accumulate the sum of squares in fp32 even in a half- or bfloat16 model; a wide sum of squares is where precision quietly dies. Wrong axis: reduce over the feature dimension only, never across batch or sequence — the latter couples tokens and leaks information at inference.

Two more. Gain initialization: initialize γ to ones so the layer starts as a pure normalizer; some codebases instead parameterize a 1 + γ gain initialized at zero, and copying weights between the two conventions without adjusting is a silent off-by-one on every channel. Expecting centering: RMSNorm does not zero the mean, so any downstream code that assumed a zero-mean activation from LayerNorm must not rely on that. None of these are hard, but each has shipped as a real bug; RMSNorm’s virtue is that there are simply fewer such traps than LayerNorm offers, because there is no mean and no bias to get wrong in the first place.