Sophia Optimizer

Why it matters

Sophia claimed AdamW improvement. Understanding shapes optimizer choice.

The architecture

Sample Hessian occasionally.

Diagonal preconditioner from Hessian.

Clip step for stability.

How it works end to end

Hutchinson estimator for diagonal Hessian.

Update Hessian every k steps.

Clip: prevent extreme steps.