Optimizer steps: SGD, momentum and Adam

Controlled quadratic experiment

0 / 60 teaching seconds

Solid blue: SGD. Dashed amber: momentum. Dotted purple: Adam.

Compare updates on one explicit objective

All three methods start at the same coordinates and minimize f(x,y) = (kx times x squared plus ky times y squared) divided by two. The exact gradient is [kx times x, ky times y]. Positive curvature makes the origin the unique minimum. There are no random gradients, batches, datasets or hidden learning-rate differences. The three trajectories use the same selected learning rate, but this does not make the setting optimal for every method. The experiment explains update rules on a small deterministic quadratic; it cannot rank optimizers for neural-network training, compare wall-clock throughput or predict generalization.

Set coordinates, curvature and hyperparameters

Start x and Start y specify coordinates between minus ten and ten. Curvature x and Curvature y specify positive coefficients from 0.1 to ten. A larger coefficient makes that direction steeper. Learning rate is the shared nonnegative step multiplier up to one. Momentum / beta1 controls the momentum accumulator and Adam first-moment decay from zero through 0.999. Adam beta2 controls its squared-gradient average through 0.9999. Adam epsilon is a positive denominator stabilizer between one trillionth and one; it is added outside the square root. Steps requests one to one hundred twenty updates. Changing any model input resets every trajectory and the inspection cursor so old optimizer state cannot leak into a new comparison.

Know the exact state conventions

Plain SGD subtracts learning rate times the current gradient. Momentum starts its buffer at zero, then adds the gradient to beta1 times the previous buffer and subtracts learning rate times that new buffer. There is no dampening, Nesterov lookahead or weight decay. Adam starts both moments at zero, averages gradients with beta1 and squared gradients with beta2, and divides each moment by one minus its decay raised to the update number. It subtracts learning rate times the corrected first moment divided by the square root of the corrected second moment plus epsilon. The inspector exposes the gradient used, position change, stored moments and corrected moments. Step zero is the initial state, before any update.

Reproduce numerical experiments

Defaults start at [2.5,2], with curvature [2,4] and learning rate 0.1. The first gradient is [5,8]. SGD and momentum both reach [2,1.2] with loss 6.88 after one update. Their second updates differ: SGD reaches [1.6,0.72], while momentum with beta1 0.9 reaches [1.15,0]. Adam moves approximately 0.1 along each initially nonzero coordinate on its first update, with the small epsilon correction retained in the exact values. Set beta1 to zero and momentum follows SGD exactly at every step. Set learning rate to zero and all positions stay fixed even though optimizer moment calculations still occur. Set both starting coordinates to zero and every method remains at the minimum.

Inspect time, geometry and failure boundaries

Play reveals the computed paths over sixty teaching seconds at normal speed; these seconds are not training duration. Pause preserves the cursor. Speed selects one quarter, one half, normal or double speed. Restart returns to step zero without altering parameters, Next step advances exactly one update, and Inspect step or Inspection time jumps to a chosen state. Reset defaults restores every parameter and normal speed. Playback starts paused, also with reduced motion enabled. Contours and all trajectories share the same coordinate transform and axis scale. Solid blue is SGD, dashed amber is momentum and dotted purple is Adam; the table also names each method so color is unnecessary. Off-plot path segments are clipped rather than falsely moved inside the square.

Interpret convergence without hiding divergence

For this diagonal quadratic, SGD converges from every starting point when the learning rate is strictly between zero and two divided by the largest curvature. At the upper boundary, a coordinate in the steepest direction alternates without shrinking. Above it, that coordinate grows unless initialized exactly at zero. Try curvature [2,4], rate 0.5 and inspect successive steps: the y coordinate alternates between plus and minus two. Increase rate to one to see growth. A method stops after the first computed coordinate exceeding one million in magnitude, retaining the boundary state and its actual update number. Other methods continue. This numerical safety cutoff is not a convergence criterion or an optimizer feature. Invalid inputs clear the plot and table and disable playback. Scroll the table on a phone for the status column; use Full precision calculation, notes and baseline capture to document the comparison.

Update-rule references

SGD and momentum equations; Adam moments and bias correction. This browser uses explicit JavaScript arithmetic rather than the PyTorch runtime.