Layer-wise LR Decay

Why it matters

Layer decay helps fine-tuning. Understanding shapes fine-tune design.

The architecture

Bottom layer: LR * decay^L.

Top layer: LR.

Common for BERT fine-tune.

How it works end to end

Decay factor: 0.65-0.95 per layer.

Bottom layer LR = peak * decay^depth.

Fine-tuning: prevents catastrophic forgetting.