Why it matters

Chinchilla revised scaling. Understanding shapes modern training.

Advertisement

The architecture

Fit L(N, D) properly.

Optimize under fixed compute.

N proportional to sqrt(C), D proportional to sqrt(C).

Chinchilla optimumFix compute CbudgetChinchilla optimumsqrt(C) each~20 tokens/paramrule of thumbModern practice over-trains: Llama 3 uses ~1000 tokens/param for inference efficiency
Chinchilla.
Advertisement

How it works end to end

Rerun Kaplan with proper HP tuning.

Chinchilla 70B > Gopher 280B on same compute.

~20 tokens/param compute-optimal.