Why it matters
Layer norm placement affects training stability and depth achievable. Modern LLMs almost universally use pre-norm; understanding why teaches about training dynamics.
The architecture
Formula: LayerNorm(x) = γ * (x - μ) / σ + β. Where μ, σ are per-token mean and std across feature dimension; γ, β are learnable scale/shift.
Differs from batch norm (which normalizes across batch dimension). Layer norm is batch-independent, works with any batch size, transfers to inference cleanly.
How it works end to end
Pre-norm placement (LayerNorm before attention/FFN, add residual after): allows very deep models (100+ layers). Standard in modern LLMs.
Post-norm placement (Attention/FFN → add residual → LayerNorm): original transformer paper. Harder to train deep models without careful init.
RMSNorm: variant using only root-mean-square without mean subtraction. Fewer parameters, similar performance, used in LLaMA.