Why it matters

LR shapes training. Understanding shapes design.

Advertisement

The architecture

Peak LR from sweep.

Scale with batch: sqrt or linear.

Model size: larger = smaller LR.

LR selection processSweep peak LRon small runScale with batchsqrt / linearAdjust model sizesmaller LR bigger modelmuP (Maximum Update Parameterization) enables LR transfer across model sizes
LR selection.
Advertisement

How it works end to end

Common LLM peak LR: 3e-4 (Llama), 6e-4 (Chinchilla).

muP: transfer from small.

Cosine decay to 10%.