Gradient Descent Convergence — Rate Analysis

Why it matters

Understanding convergence rate shapes hyperparameter tuning + expectations.

The architecture

Convex + L-smooth: O(1/k) with step 1/L.

Strongly convex: O((1-μ/L)^k) exponential.

How it works end to end

Nesterov acceleration: O(1/k²).

Non-convex: convergence to stationary point (grad = 0).

Stochastic gradient: variance affects convergence.

Adaptive methods (Adam): different theory.