Why it matters

Warmup enables stable training start. Understanding shapes training design.

Advertisement

The architecture

Linear warmup: 0 to peak over N steps.

Adam moments stabilize.

Then main schedule.

Warmup mechanicsStep 0LR = 0Warmup N stepslinear rampPeak LRmain schedule beginsAdam v_t underestimates variance early; warmup avoids too-large steps
Warmup.
Advertisement

How it works end to end

Fixes Adam's initial variance underestimate.

Prevents early divergence.

Length: 500-2000 steps or 1-3% of training.