Why it matters

Optimizer choice and learning rate schedule dominate training success. Understanding gradient descent basics matters for anyone training neural networks.

Advertisement

The architecture

Batch gradient descent: compute gradient over full dataset, take step. Impractical for large datasets.

SGD (Stochastic Gradient Descent): compute gradient on mini-batch (32-1024 samples). Standard for deep learning.

GD variantsSGDmini-batch noisy gradAdamadaptive LR per paramAdamWweight decay separateAdamW is standard for LLMs; SGD with momentum for image tasks
Optimizer variants.
Advertisement

How it works end to end

Momentum: accumulate gradient direction over time. Smooths noisy updates.

Adam: per-parameter adaptive learning rates via gradient variance estimation. Widely used.

AdamW: Adam with weight decay applied correctly (decoupled from gradient). Standard for LLM training.

Learning rate schedules: warm-up + decay. Standard for stable training.