For a small language model, the data is the model. You cannot out-scale a 100M- or 1B-parameter budget with more layers, so the lever that actually moves quality is what those parameters learn from. Two ideas govern the whole design space and pull in opposite directions. The Chinchilla result tells you how many tokens a model of a given size ‘wants’ to see to be compute-optimal — roughly 20 tokens per parameter. The inference-efficiency argument tells you to ignore that and train far past it, because a small model you will serve billions of times should be squeezed as hard as possible. Threaded through both is the Phi lesson that a curated token is worth several raw web tokens. This piece works the token-budget math from C ≈ 6ND, does a worked example for a 100M model, and then covers the data-engineering that makes each token count: exact and fuzzy deduplication, quality and perplexity filtering, domain mixing, the memorization consequences of duplicates, and where synthetic data helps and where it collapses.
Why data is the real bottleneck for an SLM
Scale laws are usually taught as ‘bigger is better,’ but a small language model is defined by the opposite constraint: the parameter count is fixed and small on purpose, so it can run on a CPU, a phone, or inside a tight latency budget. Once N is pinned, the architecture is mostly settled and the only large remaining degree of freedom is the training corpus — its size, its cleanliness, and its composition.
This is why practitioners who work on SLMs talk about data the way GPU-rich labs talk about parameters. The empirical picture is consistent: a model trained on a smaller amount of carefully filtered, deduplicated, well-mixed text reliably beats a same-size model trained on a larger pile of raw web scrape. The reason is that the loss a model reaches is set jointly by capacity and by how much signal-per-token the data carries. You cannot buy more capacity, so you buy more signal. Everything that follows — the token-budget math, dedup, filtering, mixing, synthetic data — is a way of raising the signal density of the tokens your fixed, small model is allowed to spend its learning on.
Quality over quantity: the Phi , '’': textbooks’ thesis
The sharpest statement of the quality-first view is Microsoft’s Phi line, whose first paper carried the deliberately provocative title ‘Textbooks Are All You Need.’ The Phi-1 code model was around 1.3B parameters and trained on only a few billion tokens — tiny by the standards of the day — but those tokens were either filtered to ‘textbook quality’ using a trained classifier or synthetically generated by a teacher LLM to read like clear, pedagogical explanations and worked exercises. The result punched far above its weight, and later Phi-1.5, Phi-2 (~2.7B) and Phi-3 models kept extending the recipe to general reasoning.
The mechanism to internalize is that data quality effectively shifts the entire scaling curve downward. On raw web text a model needs a certain size and token count to reach a given loss; on dense, well-explained, low-noise text it reaches the same loss with fewer parameters or fewer tokens. Curation does not repeal the scaling laws — it changes the constant in front of them. For an SLM builder with a small model and a modest compute budget, that constant is the whole game, which is why ‘spend effort on the corpus’ is the highest-leverage decision available.
Chinchilla: the compute-optimal 20:1 ratio
The counterweight to ‘just curate’ is a hard budgeting question: given a fixed training-compute budget, how should you split it between model size N and the number of training tokens D? DeepMind’s Chinchilla paper (Hoffmann et al., 2022) answered it empirically by training many models across a grid of sizes and token counts and fitting a loss surface of the form:
L(N, D) ≈ E + A / N^α + B / D^β
subject to compute C ≈ 6 · N · D (fixed)Minimizing L under the compute constraint gives the now-famous rule of thumb: the optimal number of tokens is about 20 times the number of parameters. A compute-optimal model with N parameters wants roughly D ≈ 20N training tokens. The earlier GPT-3-era intuition had badly under-trained models — 175B parameters on ~300B tokens is under 2 tokens per parameter, an order of magnitude below optimal — so Chinchilla’s headline was that most large models of the time were too big for their data. The 20:1 figure is the anchor every training-budget conversation starts from, including for small models.
The FLOPs accounting behind the ratio
The C ≈ 6ND constant deserves its own paragraph because it drives every number in this article. For a dense transformer, one forward pass costs about 2N floating-point operations per token — each parameter participates in roughly one multiply and one add. The backward pass costs about twice the forward pass, so training is ~6N FLOPs per token: 2N forward plus 4N backward.
train FLOPs C ≈ 6 · N · D (whole run)
forward/token ≈ 2 · N (inference cost)
backward/token ≈ 4 · NTwo consequences matter. First, training compute is linear in the token count D: doubling the tokens doubles the training cost, with N held fixed. Second, and crucially for SLMs, inference cost depends only on N, not on D — every generated token costs about 2N FLOPs no matter how long you trained. That asymmetry is the entire justification for over-training: you can pour arbitrarily more tokens into a small model, paying only a one-time linear training cost, and the model you deploy stays exactly as cheap to run. Training is a capital expense; inference is the recurring one.
Over-training: going past Chinchilla on purpose
Chinchilla optimizes training compute in isolation, as if you build the model once and never run it. Real SLMs are the opposite: you train once and serve the model millions or billions of times. When you account for that, the optimum shifts — the formal version is inference-aware or total-cost-optimal scaling (Sardana & Frankle, ‘Beyond Chinchilla-Optimal’). If you will generate T tokens over the model’s lifetime, total lifetime compute is roughly 6ND + 2NT, and minimizing that favors a smaller N trained on more D than the 20:1 point.
Industry practice reflects this. Llama-1 7B trained on ~1T tokens (~140:1), Llama-2 7B on ~2T (~285:1), and Llama-3 8B on ~15T tokens (nearly 1900:1) — roughly a hundred times past the Chinchilla ratio. TinyLlama pushed a 1.1B model to ~3T tokens (~2700:1). None of these are compute-optimal to train; all of them are excellent to deploy, because their creators deliberately overpaid in training to get a small, cheap, unusually capable model. For an SLM whose whole reason to exist is cheap inference, over-training is not a mistake — it is the design.
Worked example: a 100M-parameter model
Put numbers on it. Take N = 100M = 1×10^8 parameters.
Chinchilla-optimal tokens:
D* = 20 × N = 20 × 1e8 = 2e9 = 2 billion tokens
C = 6 × N × D*
= 6 × 1e8 × 2e9 = 1.2e18 FLOPs (~1.2 exaFLOP)
Over-trained to 200:1 (D = 20B tokens):
C = 6 × 1e8 × 2e10 = 1.2e19 FLOPs (10× the training cost)
Inference cost per generated token (either way):
≈ 2 × N = 2 × 1e8 = 2e8 FLOPs/token (UNCHANGED)Read the trade directly off the arithmetic. Moving from the compute-optimal 2B tokens to an over-trained 20B tokens costs 10× more training compute — because C is linear in D — and buys a noticeably stronger 100M model. The inference cost does not move at all: it is fixed at about 2e8 FLOPs per token by the parameter count. If you expect to serve, say, 10^12 tokens over the model’s life, lifetime inference is ~2e20 FLOPs — it dwarfs even the over-trained 1.2e19 training bill, so paying 10× up front to shave the recurring cost or lift quality is clearly worth it. That is the over-training calculus in one worked case.
Exact deduplication: the cheapest quality win
Before any clever filtering, remove duplicates — it is the highest return-on-effort step in the whole pipeline. Web crawls are riddled with repetition: mirrored pages, reposted articles, boilerplate headers and footers, license text repeated across millions of repositories. Left in, duplicated spans waste your token budget (you pay to train on the same sentence a thousand times) and distort the data distribution toward whatever happens to be copied most.
Exact dedup comes in two flavors. Document-level dedup hashes each document (or a normalized form of it) and drops collisions — a simple hash-set pass, O(number of documents). Substring-level dedup is more thorough: build a suffix array over the concatenated corpus and remove exactly-repeated spans above some length (say 50+ tokens), which catches the case where two otherwise different documents share a long identical passage. Lee et al. (2021), ‘Deduplicating Training Data Makes Language Models Better,’ showed this single step improves evaluation loss and sharply cuts verbatim regurgitation. For an SLM on a tight token budget, exact dedup is non-negotiable: it is free signal, recovered by deleting noise.
Fuzzy deduplication with MinHash and LSH
Exact matching misses near-duplicates: the same article with a changed byline, a template with one field swapped, lightly reworded spam. Catching those means measuring Jaccard similarity between the n-gram (shingle) sets of two documents, J(A,B) = |A ∩ B| / |A ∪ B|. Computing that for every pair is quadratic and hopeless at web scale, so the standard tool is MinHash.
For a random hash h: P( min h(A) = min h(B) ) = J(A, B)
Use K hashes → signature of K minima per document
Estimate: J_hat = (# matching signature slots) / KThe identity is exact: the probability that the minimum hash of two shingle sets agrees equals their Jaccard similarity, so a length-K signature estimates similarity cheaply. To avoid comparing all pairs, LSH (locality-sensitive hashing) splits each signature into b bands of r rows (K = b×r); documents that match on all r rows of any band land in the same bucket and become candidate pairs. The probability two documents collide is 1 − (1 − J^r)^b — an S-curve with threshold near (1/b)^(1/r). Tune b and r to set how aggressively near-duplicates get merged. This is how corpora like FineWeb and RefinedWeb dedup trillions of tokens in near-linear time.
Quality filtering with a classifier
Deduplication removes redundancy; quality filtering removes low-value text. The most common approach is classifier-based, and it is refreshingly simple. Take a set of documents you trust as high-quality — a curated reference such as Wikipedia, well-regarded books, or a hand-picked seed — as positive examples, and a random sample of raw web pages as negatives. Train a lightweight binary classifier (a fastText model or a logistic regression over hashed n-gram features is enough) to distinguish them, then score every document in the crawl and keep those above a threshold.
GPT-3 and the original LLaMA both used a variant of this recipe to filter Common Crawl. The classifier is not judging ‘truth’ — it is learning the surface statistics of prose that resembles your reference set: coherent sentences, useful vocabulary, an absence of navigation cruft and keyword spam. Two cautions. First, the filter inherits the biases of your reference set, so a narrow reference produces a narrow corpus. Second, an over-aggressive threshold throws away rare-but-valuable text (dialects, niche technical writing). Because it is cheap to run, quality classification pairs naturally with the Phi philosophy: it is the mechanical way to approximate ‘textbook quality’ across billions of documents you could never read.
Perplexity filtering
A second, complementary quality signal is perplexity. Train a small, cheap language model — classically a KenLM n-gram model — on a clean reference corpus, then score each candidate document by how surprised that model is to see it. Perplexity is the exponentiated average negative log-likelihood:
PPL(x) = exp( −(1/T) Σ_t log p(x_t | x_<t) )The heuristic used by the CCNet pipeline (which underlies much of RedPajama and other web corpora) is to keep documents in a middle band and drop the tails. Very high perplexity flags gibberish, encoding garbage, or text so off-distribution the reference model finds it incoherent. Very low perplexity is more subtle: it flags text that is too predictable — boilerplate, repeated templates, SEO filler — which is fluent but carries little information. Perplexity filtering is fast, unsupervised beyond the reference corpus, and language-agnostic, which is why it scales to hundreds of languages. Its weakness is that it rewards conventional text and can penalize legitimately unusual writing, so it works best as one vote among several rather than a sole gatekeeper.
Data mixing across domains
A corpus is not one thing; it is a mixture of domains — web prose, code, books, encyclopedic text, math, scientific papers, conversation. The mixing weights — what fraction of training tokens comes from each — materially change the resulting model, and the naive choice (sample each domain in proportion to how much of it you happened to crawl) is almost never optimal. Web scrape is abundant but noisy; code and math are scarcer but disproportionately useful for reasoning, so they are usually upweighted relative to their raw share.
The Pile made these weights explicit, hand-assigning 22 sources with per-source multipliers. More systematic methods learn the weights: DoReMi (Xie et al.) trains a small proxy model to find domain weights that minimize worst-case loss across domains, then reuses those weights to train the real model, improving both perplexity and downstream accuracy. For an SLM the mixing decision is amplified precisely because capacity is scarce: with only 100M parameters you cannot afford to spend a large fraction of them modeling low-value boilerplate, so tilting the mixture toward dense, high-signal domains — and holding out a little code and math even for a general model — is one of the cheapest quality levers you have.
Deduplication and memorization
Deduplication is not only a quality tactic; it directly controls how much the model memorizes. The empirical law, established by Carlini et al. (‘Quantifying Memorization’) and Kandpal et al., is blunt: the probability that a model reproduces a training sequence verbatim grows sharply — roughly log-linearly — with the number of times that sequence appeared in the training set. A string seen once is unlikely to be regurgitated; a string duplicated a thousand times is very likely to be.
This ties the whole pipeline together. Removing duplicates does three things at once: it recovers token budget, it improves evaluation loss, and it reduces memorization — which matters for privacy (verbatim leakage of personal data), for copyright exposure, and for honest benchmark numbers (a model that memorized a test-adjacent document scores misleadingly well). It also has a practical training corollary: because de-duplicated data provokes less memorization per exposure, you can safely train for more epochs over a clean corpus before over-fitting — exactly the regime an over-trained SLM lives in. Dedup is thus both a quality step and a safety step, and the two reinforce each other.
Repeating data: how many epochs are safe?
Over-training a small model on a limited high-quality corpus raises an unavoidable question: what happens when you run out of unique tokens and start repeating them? If curated data is finite — and good data always is — you either dilute it with lower-quality tokens or you loop over it for multiple epochs. Muennighoff et al. (‘Scaling Data-Constrained Language Models’) studied this directly.
Their finding is encouraging and practical: repeating data for up to about 4 epochs is nearly as good as having that many fresh tokens — the marginal loss reduction per repeated epoch stays close to the fresh-token curve. Beyond roughly 4 epochs, returns diminish quickly, and past a dozen or so, extra epochs add essentially nothing while increasing memorization risk. The operating rule for a data-constrained SLM follows: assemble the best corpus you can, deduplicate it, and cycle over it a handful of times rather than either stopping early at the Chinchilla point or padding the run with junk tokens to hit a token target. A few clean epochs beat one epoch of clean-plus-garbage, and both beat a single under-trained pass.
Synthetic data: promise and collapse
The Phi results put synthetic data at the center of the SLM playbook. The idea is to use a strong teacher model to manufacture exactly the dense, well-explained text you wish the web had — textbook-style explanations, graded exercises with worked solutions, question-answer pairs, or rephrasings of raw web pages into cleaner prose (the ‘Rephrasing the Web’ / WRAP approach). Because you control the generator, you can target coverage, difficulty, and style, and you can produce reasoning-dense tokens that are rare in natural scrape. For a small model this is leverage: it is the most direct way to raise signal-per-token.
The danger is model collapse. Shumailov et al. showed that training recursively on a model’s own outputs degrades the tails of the distribution — rare events vanish, diversity narrows, and successive generations get blander and more error-reinforcing. There is also contamination risk: a teacher can leak benchmark answers into your training set. The discipline that keeps synthetic data useful is to ground it in real seeds, use a genuinely stronger teacher rather than a same-strength model, deduplicate and filter the synthetic output like any other source, and keep it as a complement to real human text rather than a replacement for it.
Assembling a pipeline for a CPU-scale SLM
Stack the pieces in the order that spends effort where it pays. Start from a broad but reasonable source (a filtered web corpus, plus code, books, and math). Deduplicate first — exact document and substring dedup, then MinHash/LSH for near-duplicates — because everything downstream is cheaper on a smaller, cleaner set. Then quality-filter with a classifier and a perplexity band to strip spam and gibberish. Then mix the surviving domains with deliberate weights, upweighting dense sources. Optionally add curated synthetic data to fill gaps in reasoning-heavy content.
Finally, size the run against the math. Compute the Chinchilla anchor (~20N tokens) as a floor, decide how far past it to over-train given your expected inference volume, and if your clean corpus is smaller than that target, loop it for a few epochs rather than diluting it. One more engineering note from the stub worth keeping: pre-tokenize once to a binary token file and mmap random spans at train time, so tokenization never sits on the critical path. The whole pipeline is a single sustained bet — that for a small model, a smaller number of clean, well-mixed, hard-won tokens beats a larger number of raw ones — and every stage above is one more way to make that bet pay.
N (about 2N FLOPs per token) while training cost is linear in D (6ND), a model you will serve billions of times should be over-trained far past 20:1, exactly as Llama and TinyLlama do. A worked 100M model wants ~2B tokens to be compute-optimal, but spending 10× the training compute on 20B tokens leaves inference untouched and buys a better model. The tokens themselves must earn their place: deduplicate exactly and with MinHash/LSH, quality-filter with classifiers and perplexity bands, mix domains with intent, and add synthetic data carefully to avoid collapse. Deduplication doubles as a memorization and privacy control, and a clean corpus can safely be repeated for a few epochs. The through-line is the Phi lesson: a curated token is worth several raw ones, and for a fixed, small model that multiplier is the whole game.