Why it matters
Activation scaling shapes stability. Understanding shapes deep training.
Advertisement
The architecture
LayerNorm / RMSNorm control layer output.
Residual: identity + delta.
Init controls delta scale.
Advertisement
How it works end to end
Pre-LN vs Post-LN.
Residual gain init.
Careful across depth.
Signal propagation theory.