Why it matters

Chinchilla changed how models are trained. Understanding shapes model design.

Advertisement

The architecture

Fit L(N, D) with power law.

Optimize under compute constraint.

Optimal N proportional to D.

Chinchilla optimumFit power lawover model runsCompute-optimalN and D scale equally~20 tokens per paramrule of thumbModern models often overtrain by 10-100x tokens per param for inference efficiency
Chinchilla.
Advertisement

How it works end to end

Chinchilla optimum: N proportional to D^0.5 approximately.

Approx: 20 tokens per param.

Modern practice: over-train for inference efficiency (Llama 3 = 1000+ tokens per param).