Scaling Laws Deep Dive

Why it matters

Scaling laws guide training investment. Compute is expensive; laws optimize spend.

The architecture

Loss ~ N^-a + D^-b + irreducible.

Chinchilla: balanced N and D optimal.

How it works end to end

Kaplan 2020.

Chinchilla 2022.

Balanced N + D optimal.