INITIALISATION · KEEPING ACTIVATIONS IN RANGE
Weight Initialization Distributions
Push a batch of random inputs through a stack of freshly initialised layers and watch whether the activations keep their scale, vanish or explode.
Your experiment
Start with Xavier weights, ReLU, 24 layers of width 128 and no residual connection. Read the gain per layer, then try He, the fixed 0.02, tanh and GELU, add a plain residual connection and then a scaled one, and switch the distribution to uniform.
Every input recomputes the result immediately; there is no animation because nothing here unfolds over time. An input outside its allowed range is rejected with a message and the previous valid result stays on screen.
Computed data
Metrics
Each layer is a dense square matrix with normal weights of the chosen standard deviation and no bias, applied to 16 inputs with independent standard normal entries; the activation RMS is measured after every layer. VANISHES means a final RMS below 0.01 and EXPLODES above 100, thresholds chosen for readability. Real transformers add normalisation layers that rescale activations, so this shows why the initial scale matters, not how a full model trains.
Why the scale matters
Each layer multiplies its input by a random matrix. If the weights have standard deviation s and the layer has n inputs, each output is a sum of n terms and its variance is about n x s squared times the input variance, before the activation. Keep n x s squared near the right constant and activations keep their size through many layers; miss it and they shrink or grow geometrically. That is what Xavier, He and LeCun initialisation are designed to prevent. Only the variance matters for this, not the shape: set Distribution to uniform and the histogram shows the first layer's weights, and uniform weights with the same standard deviation stay within 1.732 standard deviations, a largest weight of 0.1531 here, while 4.78% of normal weights lie beyond 2, yet both vanish at almost the same rate, 0.6992 and 0.696 per layer.
ReLU halves the variance
The Weight standard deviation control picks the scale rule. The opening stack uses Xavier weights, std 0.08839 for width 128, and ReLU. ReLU zeros about half of its inputs, 50.6% after layer 1, so each layer keeps only about 1 / sqrt(2) of the RMS: the measured gain is 0.6992 per layer, and after 24 layers the RMS is 1.84e-4: VANISHES. He initialisation, std 0.125, doubles the variance to compensate: gain 0.9889 and RMS 0.7535 after 24 layers, 0.716 after 48. Because the layers are square, LeCun gives the same std as Xavier here.
Activation functions differ
With tanh, which is close to linear near zero, Xavier keeps the activations in range: RMS 0.133 after 24 layers. With no activation at all the gain is 0.9991. GELU behaves more like ReLU: Xavier vanishes to 1.16e-7, and even He weights lose scale slowly, 0.0394 after 24 layers. The fixed standard deviation of 0.02 used in GPT-2 shrinks the RMS by 0.1582 per layer here and reaches 5.97e-20, because without normalisation nothing restores the scale.
Residual connections grow the stream
A residual connection adds each layer's output to its input, x + f(x). The variances add instead of multiply, so nothing vanishes, but the stream grows: with Xavier weights the RMS reaches 5804.5254 after 24 layers, about 1.4358 per layer: EXPLODES. Dividing each branch by the square root of the number of layers, the idea behind GPT-2's scaled residual initialisation, keeps the final RMS at 3.6332.
Reading the tool and its limits
The bars show the share of first-layer weights in half-standard-deviation bins, the line chart plots log10 of the activation RMS per layer against the input scale, the lanes show the standard deviation, std x sqrt(fan-in) and the gain, and the table lists selected layers with the ratio to the previous layer and the share of zeros. Everything is computed from seeded random numbers, so changing the Random seed draws new weights and inputs; it changes the details but not the trend. Layer normalisation, attention and training dynamics are not modelled.