Why it matters

Compute-optimal guides research. Inference cost may favor smaller/longer-trained.

Advertisement

The architecture

C ≈ 6ND (forward + backward).

Optimize N vs D under C.

Inference favors smaller.

Compute-optimal trainingBudget Cfixed computeC = 6NDflops ruleOptimal N, DbalancedChinchilla suggests balance; inference favors smaller + more tokens
Compute-optimal training.
Advertisement

How it works end to end

C ≈ 6ND.

Chinchilla point.

Inference favors smaller.