Why it matters
RMSNorm is default modern LLM norm. Understanding shapes design.
Advertisement
The architecture
y = x / sqrt(mean(x^2) + eps) * gain.
No bias term.
Advertisement
How it works end to end
Introduced by Zhang and Sennrich 2019.
Simpler than LayerNorm.
Slightly faster; equal quality.