Compute-Optimal Training

Why it matters

Compute is the real constraint. Understanding shapes model planning.

The architecture

Compute C = 6 N D (rough).

Chinchilla: N proportional to sqrt(C), D proportional to sqrt(C).

How it works end to end

FLOP approximation: 6 N D forward+backward.

Data: needs to be available at chosen D.

Overtraining: inference wins.