Why it matters

LR schedule shapes training. Understanding shapes design.

Advertisement

The architecture

Warmup: linear from 0 to peak over N steps.

Decay: cosine or linear to final LR.

LR scheduleWarmuplinear ramp 500-2000 stepsPeak LRconstant brieflyDecaycosine to 10% of peakCosine decay + linear warmup dominant; some use WSD (warmup-stable-decay)
LR schedule.
Advertisement

How it works end to end

Warmup: 500-2000 steps typical.

Cosine: L(t) = final + (peak - final) * (1 + cos) / 2.

Continue past cosine end at final LR.