Learning-rate schedule: warmup and exact decay

Set the schedule

0 / 60 teaching seconds

The cursor inspects a schedule. Playback seconds are not training time.

Primary reference: PyTorch cosine annealing equation and iteration convention.

Define the step convention

This lab computes a learning-rate multiplier from an explicit step index. It does not train a network or generate a loss curve. Step zero is the schedule value before advancing the schedule; Total steps is an included endpoint. An optimizer that uses indices zero through total minus one will not use the endpoint value for an update. Whether a framework calls its scheduler before or after an optimizer update changes which value is used first. Match that convention before copying the numbers into a training loop. The curve has a linear warmup followed by one decay segment without restarts.

Choose the schedule inputs

Total steps accepts an integer from two to one million. Warmup steps accepts zero through total minus one, leaving at least one decay interval. Peak learning rate is a positive number at most one; it scales every computed rate without changing the normalized curve. Final / peak ratio accepts zero through one and controls the endpoint relative to the peak. Decay rule selects cosine or linear interpolation. These are schedule parameters, not measured recommendations for any model. Changing a parameter pauses playback and returns inspection to step zero. Reset defaults restores ten thousand total steps, one thousand warmup steps, a peak of 0.001, a final ratio of 0.1, cosine decay and normal playback speed.

Compute the value at a chosen step

During warmup, the multiplier is step divided by warmup steps. At the warmup boundary it is exactly one. After warmup, let u equal the elapsed decay steps divided by total minus warmup. Linear decay uses final ratio plus one minus final ratio times one minus u. Cosine decay replaces that last factor with one half times one plus cosine of pi times u. Multiply the result by Peak learning rate. The full-precision calculation and numerical table expose these values directly; displayed decimal values are rounded. The blue chart is normalized by the peak, so raising the peak changes numeric rates while keeping its shape.

Reproduce boundaries and distinguish decay rules

With defaults, inspect step zero for rate zero, step 500 for 0.0005, step 1000 for 0.001, step 5500 for 0.00055 and step 10000 for 0.0001. At step 3250 the cosine multiplier is approximately 0.868198052, while linear decay gives 0.775. Both rules agree at the midpoint and endpoints, so inspecting only those points cannot distinguish them. Set Warmup steps to zero: the first value becomes the peak immediately and no division by zero occurs. Set Final / peak ratio to one: decay is flat at the peak, but any positive warmup still ramps upward. Set the ratio to zero to end at exactly zero. A warmup equal to total is rejected rather than silently producing an undefined decay interval.

Inspect without confusing playback with computation

Inspect step selects any integer schedule index, including the endpoint, and pauses playback. Play moves the inspection cursor through the already-defined schedule over sixty teaching seconds. Pause freezes it; Restart returns to step zero while preserving parameters and speed. Speed offers quarter, half, normal and double speed. Inspection time can seek anywhere on that timeline and rounds to the nearest schedule step. A complete run therefore lasts at least thirty seconds at the fastest setting. No animation starts automatically, including when reduced motion is requested. The table includes start, warmup midpoint, warmup boundary, decay midpoint, endpoint and the inspected step, removing duplicate indices. Invalid inputs clear the table and chart and disable Play.

Keep the conclusions within the model

This arithmetic does not predict convergence, suitable warmup duration, stable peak rate or final validation quality. Gradient accumulation may make optimizer updates less frequent than data batches; Total steps must use the intended update unit consistently. A cosine curve is not evidence that an optimizer follows a cosine path in parameter space. Numerical outputs use JavaScript double precision, and the chart samples two hundred intervals; the selected step is calculated directly even when it falls between plotted samples. The reference links describe related scheduler conventions, but this page states its own endpoint and warmup rules explicitly.