Why it matters

Double descent explains why huge models don't overfit as classically predicted.

Advertisement

The architecture

Under-parameterized: classic bias-variance.

Interpolation: peak error.

Over-parameterized: error drops again.

Double descentSmall modelclassic U-curveInterpolation peakhighest test errOverparameterizederror dropsLLMs live in overparameterized regime; supports 'bigger is better'
Double descent.
Advertisement

How it works end to end

Discovered by Belkin et al. 2018.

Empirically robust across architectures.

Explains huge models' success.