CALCULATE · INSPECT · COMPARE
LayerNorm and RMSNorm: inspect row statistics
Compare mean-centering with root-mean-square scaling using the same input.
Your experiment
Turn on the constant-row fixture. LayerNorm without learned bias returns zeros; RMSNorm keeps a nonzero row when the input is nonzero. Set scale to zero: both are zero and remain finite because epsilon is positive. Change epsilon and inspect the denominator.
Play inspects the entire calculated trace over 60 seconds at 1×. Use 0.5× for two minutes, pause, or step through individual operations. Restart preserves your inputs. Changing a model input rebuilds the calculation without autoplay.
What the calculation means
LayerNorm uses (x − mean)/sqrt(population variance + epsilon). RMSNorm uses x/sqrt(mean(x²) + epsilon). Both normalize each row over its full width. Learned gain/bias and reduced-precision rounding are excluded. Epsilon makes the output variance or RMS slightly less than one.
Matrix values at this step
Canvas and tables round labels for readability. The inspector preserves full JavaScript number values.
Full-precision operation inspector
A bounded mathematical teaching model. Playback measures inspection time, not computation or training latency. No model weights are downloaded; no external inference, storage or training is performed. Read the official AICassindra articles.
Two row statistics, one explicit denominator
Normalization operates independently on each token row. LayerNorm subtracts that row's mean and divides by the square root of population variance plus epsilon. RMSNorm divides the original row by the square root of its mean square plus epsilon, without subtracting the mean. The Statistics table contains mean, variance or mean square, then denominator, in that order. This experiment fixes learned gain to one and bias to zero; it does not simulate a trained normalization layer or its gradients. Population variance divides by the row width, not by width minus one.
Generated matrices and explicit input
Input rows sets the number of independent token rows. Input width sets the features in each row. Matrix fixture seed deterministically regenerates small signed integers, so record it with every comparison. Generated input scale multiplies the generated input values, including negative or zero scaling. Open Edit the input matrix to enter a rectangular JSON array matching the selected row and feature counts. An explicit matrix overrides generated input values and input scaling, but the seed still determines any generated weight matrices. Clear the editor to restore the generated fixture. Malformed JSON, a mismatched shape or any component outside minus one hundred through one hundred disables calculation and clears stale results.
Constant rows expose the difference
Select two rows and width three, clear the input editor, set Generated input scale to one and enable Use constant generated rows. Each row contains [2,2,2]. With LayerNorm, the mean is two, variance is zero and all outputs are zero. Switch Normalization to RMSNorm: mean square is four and every output is 2 divided by sqrt(4 + epsilon), close to one. The matrix seed no longer changes this constant input. Disable the constant checkbox to restore seeded values. The explicit JSON editor takes priority over the constant fixture as well as generated scaling.
Epsilon changes small-signal behavior
Choose epsilon 0.1, enter [[-1,0,1],[-1,0,1]] and compare the two modes. Since each row has mean zero, both produce the same output: [-1,0,1] divided by sqrt(2/3 + 0.1). Now enter [[0,1,2],[0,1,2]]. LayerNorm is unchanged by that shared additive shift; RMSNorm is different because mean square becomes 5/3. Try all epsilon choices on a nearly constant row and inspect the denominator. Epsilon is inside the square root and prevents division by zero. Exact unit variance or unit root mean square is not promised with positive epsilon.
Interpret the boundaries honestly
Zero input produces zero output in both modes. Negative generated scaling can flip output signs; the positive epsilon means rescaling invariance is only approximate at small amplitudes. Changing row count does not mix row statistics. Changing width changes the population over which statistics are calculated. This is an arithmetic comparison, not evidence that one method trains better or runs faster on a particular device. Rounded display values may conceal small differences; use the full-precision inspector to reproduce the experiment numerically.
Inspect computation at your own pace
Play spreads the finite operation trace over sixty seconds at normal speed. Pause freezes the arrays and accumulator so you can compare a highlighted input with its contribution. Next operation advances one recorded calculation and pauses. Restart returns to the first frame while preserving the current experiment inputs; it is not a factory reset. Speed selects a quarter, half, normal or double inspection rate. The Inspection slider directly selects a point in that minute and pauses playback. Editing any model parameter rebuilds the trace at its beginning without autoplay. These are viewing times, not CPU performance measurements.