AdamW Math Deep Dive

Why it matters

AdamW is default modern LLM optimizer. Understanding shapes training.

The architecture

Compute m, v (moving averages).

Step = lr * m_hat / (sqrt(v_hat) + eps).

Apply weight decay separately: w = w - lr * wd * w.

How it works end to end

Loshchilov + Hutter 2019.

Betas: 0.9, 0.95 for LLMs.

Weight decay: 0.1 typical.

Epsilon: 1e-8.