Chinchilla Scaling

Why it matters

Chinchilla revised training recipes. Under-trained models waste compute.

The architecture

N params + ~20N tokens.

Loss minimized at balanced.

70B trained on 1.4T.

How it works end to end

Hoffmann 2022.

~20 tokens/param.

Chinchilla 70B / 1.4T.