Why it matters

Depth scaling shapes model design. Understanding shapes strategy.

Advertisement

The architecture

More layers = more compute per token.

Init: down-scale residuals.

muP variants for depth transfer.

Depth scaling stackDeeper stackmore layersDepth muPinvariant HPsResidual downscale1/sqrt(L)Cerebras + Bengio depth muP variants; complements width muP
Depth scaling.
Advertisement

How it works end to end

Residual downscale: 1/sqrt(L) or 1/L.

Depth muP: LR transfer with depth.

Init: layer-index-dependent.