Why it matters

LR transfer is highest-value HP transfer. Understanding shapes design.

Advertisement

The architecture

muP: LR width-invariant.

Non-muP: LR ~ 1/sqrt(width).

Depth: further adjustment.

LR transfer flowSmall modelLR sweepTransfer rulemuP or 1/sqrt(w)Large model LRpredictedEven without muP, empirical LR ~ 1/sqrt(width) works surprisingly well
LR transfer.
Advertisement

How it works end to end

muP LR: constant across widths.

Non-muP: scale as 1/sqrt(width).

Depth: further tune.

Cerebras muP variants.