Why it matters

Optimizer choice shapes training cost + quality. Understanding shapes design.

Advertisement

The architecture

SGD: simple; requires careful LR.

Adam: adaptive per-param; needs Beta1, Beta2.

AdamW: decoupled weight decay.

Optimizer familiesSGDsimple gradient stepAdam / AdamWadaptive momentSecond-orderShampoo, K-FACAdamW standard for LLMs; second-order methods emerging at scale
Optimizers.
Advertisement

How it works end to end

Adam: m_t = beta1 m_(t-1) + (1-beta1) g; v_t similar with g^2.

Step: m_hat / (sqrt(v_hat) + eps).

AdamW: weight decay separate from gradient.