Why it matters

Muon challenges AdamW dominance. Understanding shapes optimizer trends.

Advertisement

The architecture

Compute momentum M.

Newton-Schulz iteration: orthogonalize M.

Step: orthogonalized momentum.

Muon optimizerMomentum Mbeta accumulationNewton-SchulzorthogonalizeStepwith ortho momentumMuon (Jordan et al. 2024) reduces training time on GPT-2 scale by 30%
Muon.
Advertisement

How it works end to end

Newton-Schulz iteration: X_{t+1} = 1.5 X_t - 0.5 X_t X_t^T X_t.

Approximate orthogonalization.

Faster per-token training.