Why it matters

Cosine is default decay for LLM. Understanding shapes schedule.

Advertisement

The architecture

LR(t) = final + (peak - final) * 0.5 * (1 + cos(pi * progress)).

Progress: 0 to 1 over decay period.

Cosine decay curveProgress 0LR = peakProgress 0.5midwayProgress 1.0LR = finalSmooth end; some use WSD to avoid decay committing early
Cosine.
Advertisement

How it works end to end

Peak LR for main training.

Final LR = 10% peak typical.

Smooth curve prevents shock.