Why it matters

Data is often the bottleneck. Understanding shapes data strategy.

Advertisement

The architecture

Quality: dedup + filter + curate.

Quantity: 15T+ tokens for modern SOTA.

Diversity: domain + language mix.

Data scalingCurate + dedupquality filterScale up15T+ tokensDiversifydomains + languagesRecent frontier: Llama 3 15T, DeepSeek 15T; quality curation increasingly key
Data scaling.
Advertisement

How it works end to end

Curation: dedup, safety, language.

Domain mix: code, math, papers, web.

Synthetic augmentation.

Filter models: heuristic + ML-based.