Why architecture matters here

Scaling-law mistakes waste money at scale. Architecture matters because C, N, D are joint knobs; picking wrong wastes compute or produces undertrained models.

Advertisement

The architecture: every piece explained

The top strip is the primitives. Compute budget C. Parameter count N. Data tokens D. Loss law L(N, D) fitted.

The middle row is regimes. Chinchilla N ≈ D. Compute-optimal. Overtrain regime for cheap serving. Inference frontier.

The lower rows are practice. Data quality. Emergent capabilities. Ops — budget picking + trade + planning.

Scaling laws — Chinchilla + compute-optimal + data budget + inference frontierhow big and how much dataCompute budget CFLOPs availableParameter count Nmodel sizeData tokens Dtraining corpusLoss lawL(N, D) fittedChinchillaN ∝ D roughly equalCompute-optimalmin loss at COvertrain regimesmall N, huge DInference frontierserving cost vs qualityData qualitymatters beyond countEmergent capabilitiesat scaleOps — budget picking + trade + planningbalancefindshiftserveboostunlockunlockoperateoperate
Scaling laws: parameters × data × compute → loss.
Advertisement

End-to-end flow

End-to-end: team has 1e24 FLOP budget. Chinchilla suggests ~30B params with ~600B tokens. Team overtrains a 7B on 3T tokens instead because inference cost matters more than pretraining cost. Loss slightly worse than optimal, but inference 4x cheaper.