Why it matters

Pretraining scaling shapes model capability. Understanding shapes strategy.

Advertisement

The architecture

Chinchilla optimum baseline.

Over-train for inference (Llama 3 approach).

Data curriculum + quality filters.

Pretraining scaling leversChinchilla baselinecompute-optimalOver-train optionfor inferenceData + curriculumquality curationFrontier: over-train + high-quality data + long context; Llama 3, DeepSeek V3 examples
Pretrain scaling.
Advertisement

How it works end to end

Over-training: 10-100x Chinchilla.

Quality data > quantity.

Curriculum: math + code + web + papers.