Why it matters

muP saves massive sweep cost. Understanding shapes modern LLM training.

Advertisement

The architecture

Rescale init + LR + attention scale by width.

Ensures updates same magnitude.

Tune HPs at small width; use at large.

muP HP transfer flowSmall model tunecheap HP searchmuP scalingwidth-independent HPsLarge modelsame HPsYang et al. 2021; saves months of frontier LLM hyperparameter search
muP.
Advertisement

How it works end to end

Yang + Hu 2021.

Init: fan-in scaling.

LR: per-layer rescaled.

Attention: 1/d instead of 1/sqrt(d) in specific places.