Synthetic Data Pipelines, in depth: a cost-ordered funnel of verifiers, dedup, judges and decontamination

Why this needs architecture

Synthetic data fails silently and late. A serving bug pages someone tonight; a bad training batch surfaces weeks later as an eval score that is suspiciously high, because benchmark questions leaked into training, or flat, because 100,000 near-identical examples taught the model one thing 100,000 times. By then the batch has been shuffled into a larger mix and, without lineage, nobody can say which examples to remove.

Cost decides the shape. A regex check costs microseconds per example, an execution check milliseconds, an embedding lookup a little more, an LLM judge a model call, and a training run to learn whether the batch helped costs GPU hours. So the pipeline is a funnel ordered by cost per example: each stage should remove everything a cheaper stage could have caught, so the expensive stages see only plausible candidates.

Why this needs architecture

Synthetic data fails silently and late. A serving bug pages someone tonight; a bad training batch surfaces weeks later as an eval score that is suspiciously high, because benchmark questions leaked into training, or flat, because 100,000 near-identical examples taught the model one thing 100,000 times. By then the batch has been shuffled into a larger mix and, without lineage, nobody can say which examples to remove.

Cost decides the shape. A regex check costs microseconds per example, an execution check milliseconds, an embedding lookup a little more, an LLM judge a model call, and a training run to learn whether the batch helped costs GPU hours. So the pipeline is a funnel ordered by cost per example: each stage should remove everything a cheaper stage could have caught, so the expensive stages see only plausible candidates.

Diversity is designed at the prompt stage

No filter can add information that generation never produced, so diversity has to be designed in at the prompt stage. Raising temperature mostly adds noise; a generator left to defaults writes a thousand variations on its five favourite examples. The reliable lever is an explicit coverage grid: a taxonomy of what the data must span, crossed into prompts, with a few real seed examples per prompt as style anchors. The grid also becomes the unit you rebalance later, because every example remembers which cell it came from.

import itertools, random

personas   = load("personas.txt")        # e.g. 40 analyst / engineer / student profiles
topics     = load("schema_topics.txt")   # e.g. 200 database schemas
difficulty = ["single-table", "join", "aggregate+join", "window+subquery"]

def build_prompts(seeds, k_seeds=3, rng=random.Random(7)):
    for persona, topic, level in itertools.product(personas, topics, difficulty):
        shots = rng.sample(seeds[level], k_seeds)          # real, vetted examples
        yield {
            "cell": (persona, topic, level),               # kept for lineage + quotas
            "prompt": TEMPLATE.format(persona=persona, topic=topic,
                                      level=level, examples=shots),
            "seed_ids": [s["id"] for s in shots],
        }

Verifiers before judges

Wherever the task has a checkable answer, check it mechanically before any model judges it. Generated SQL can be executed against the schema and its result compared with the claimed answer. Generated code can run against generated tests, which themselves must be checked against a reference implementation, or they simply agree with the bug. Math answers can be parsed and compared numerically. These verifiers are cheap, deterministic and immune to the persuasive-but-wrong outputs that fool model judges.

import sqlite3

def verify_sql(example, db_path):
    con = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)   # read-only copy
    con.set_progress_handler(lambda: 1, 1_000_000)              # abort runaway queries
    try:
        rows = con.execute(example["sql"]).fetchall()
    except sqlite3.Error as e:
        return False, f"error: {e}"
    finally:
        con.close()
    return normalise(rows) == normalise(example["claimed_result"]), "ok"

Run verifiers in a sandbox with time and memory limits. Generated code is untrusted input, and a pipeline that executes it on a shared host is a remote-code-execution service you built yourself.

Deduplication: lexical and semantic

Deduplication runs in two passes. The first removes exact and near-exact copies with MinHash and locality-sensitive hashing, which compares compact signatures instead of every pair. The second works in embedding space: cluster the survivors and cap how many examples one semantic neighbourhood may contribute. The second pass is the real defence against mode collapse in the dataset, because a generator's favourite pattern reappears in different words, which MinHash does not catch.

from datasketch import MinHash, MinHashLSH

def signature(text, n=5):
    m = MinHash(num_perm=128)
    toks = text.lower().split()
    for i in range(max(1, len(toks) - n + 1)):
        m.update(" ".join(toks[i:i + n]).encode("utf8"))
    return m

lsh = MinHashLSH(threshold=0.8, num_perm=128)   # ~Jaccard 0.8 over word 5-grams
kept = []
for ex in candidates:
    sig = signature(ex["text"])
    if lsh.query(sig):            # a near-duplicate is already kept
        continue
    lsh.insert(ex["id"], sig)
    kept.append(ex)

Judge panels and calibration

Model judges score what survives on a written rubric: is the response correct, does it follow the instruction, is it safe, is the reasoning actually used to reach the answer. A single judge has blind spots and preferences, including a known bias towards longer answers and towards outputs from its own model family, so use two or three judges or prompts and require agreement. Calibrate them before trusting them: hand-label a few hundred examples, measure each judge's agreement with humans, and re-measure whenever you change a judge model or prompt. Store raw scores, not just pass or fail, so thresholds can move later without regenerating.

Decontamination as a hard gate

Decontamination is a hard gate, not a score. Any example that overlaps an evaluation set you report on must go, or your evals measure memorisation. Generators make this worse than with scraped data, because a strong model may have memorised public benchmarks and reproduce them nearly verbatim. Check long n-gram overlap (13-grams are a common choice) and also embedding similarity, since paraphrased benchmark items slip past n-grams.

def ngrams(text, n=13):
    t = text.lower().split()
    return {" ".join(t[i:i + n]) for i in range(len(t) - n + 1)}

eval_index = set()
for item in all_eval_sets():               # every benchmark you will ever report
    eval_index |= ngrams(item["text"])

def contaminated(example):
    return not ngrams(example["text"]).isdisjoint(eval_index)

Lineage on every example

Every example that survives carries a lineage record. Without it you cannot recall a bad batch, rebalance a cell, or answer where a strange model behaviour came from. Keep it small and attach it to the example, not to a separate log that drifts out of sync.

{
  "id": "sql-2026-10-04-000731",
  "batch": "sql_reasoning_v3",
  "cell": ["data analyst", "retail_orders", "window+subquery"],
  "template": "sql_cot_v7", "seed_ids": ["s-112", "s-418", "s-903"],
  "generator": {"model": "gen-model-x", "temperature": 0.8, "sample": 2},
  "verifier": "sqlite_exec_v2:pass",
  "judges": {"judge_a": 4.5, "judge_b": 4.0},
  "decontam": {"ngram13": "clean", "embed_max_sim": 0.71}
}

Worked example: 50,000 SQL reasoning examples

Suppose you need 50,000 multi-step SQL reasoning examples. The templater crosses 40 personas, 200 schema topics and 4 difficulty tiers into 32,000 prompts, and the generator writes 4 candidates per prompt.

The yield is 50,000 / 128,000, about 39 percent. Two numbers matter more than the yield. The judges saw 67,000 examples, not 128,000, so ordering saved nearly half the judge calls; with three judges that is 183,000 avoided calls. And the semantic cap removed more than the MinHash pass, which says the generator was collapsing onto a few patterns. The fix belongs upstream: add difficulty or schema variety to the grid rather than filtering harder. Finally, fine-tune a small model on the batch, compare it with a baseline on held-out SQL evals, and ship only if it wins.

The ablation gate: proving a batch helps

Filters tell you an example looks good; only training tells you a batch helps. The ablation gate is a small, fixed experiment run for every candidate batch: take a frozen base checkpoint, fine-tune it once on the current production mix and once on the mix plus the new batch, with the same steps, learning rate and seed, then evaluate both on held-out sets that the decontamination index protects. A LoRA adapter on a small model is often enough to see the direction of the effect at a fraction of full-training cost.

Decide the acceptance rule before you see results: for example, the target skill must improve by more than run-to-run noise, measured by repeating the baseline with two or three seeds, and no guarded capability may regress by more than a set margin. A batch that improves its own skill while quietly damaging instruction following or safety refusals is a rejection, not a trade-off to argue about later.

Running the pipeline

Run the funnel as a batch DAG with idempotent stages. Each stage reads a versioned input shard, writes a versioned output shard plus a small rejection log with reasons, and can be re-run without touching other stages. That makes it cheap to change a judge threshold or a dedup setting and recompute from the stored candidates instead of paying for generation again. Cache generator calls by a hash of prompt, model and sampling parameters so a crashed job resumes rather than regenerates.

Track three dashboards per batch: per-stage removal counts and reasons, per-cell counts against quotas, and cost per surviving example. A sudden change in any removal rate usually means something upstream changed, such as a new generator version or a template edit, and is worth investigating before the batch goes near training.

Model collapse and the mix

Training repeatedly on a model's own outputs narrows the distribution: rare cases thin out generation by generation, an effect Shumailov and colleagues described as model collapse. Follow-up work found that accumulating synthetic data alongside the original real data, rather than replacing it, largely avoids the degradation. The practical rules are to keep real data in every mix, set synthetic-to-real ratios per domain rather than globally, and watch tail metrics such as rare-class recall, not just averages.

Also check the terms of the generator you use. Some model providers restrict using outputs to train competing models, and data containing personal information keeps its privacy obligations after a model has paraphrased it.

Failure modes

Trade-offs

Synthetic data is strongest where correctness is checkable (code, SQL, math, tool calls, structured extraction) and weakest where truth depends on facts the generator may not know, where it will invent plausible details. Distilling from a stronger model is usually cheaper than human labelling but caps the student near the teacher on that task. Human-written seeds and human review of a sample remain worth their cost, because they anchor the taxonomy and calibrate the judges.

Read on: proving fidelity, utility and privacy, Self-Instruct, Evol-Instruct and input diversity, calibrating LLM judges, filtering, dedup and mixture design and model distillation.

What to do next

1. Write down the eval sets you report on and build the decontamination index before generating anything.
2. Define a coverage grid and collect real seed examples for every cell.
3. Add a mechanical verifier for every task type that has a checkable answer, and sandbox it.
4. Order the filter stack by cost per example and log how many examples each stage removes.
5. Run MinHash dedup and a semantic cap; if the cap removes a lot, fix the grid, not the threshold.
6. Calibrate two or three judges against a few hundred human labels.
7. Attach a lineage record to every surviving example.
8. Gate each batch with a small fine-tune against a baseline, and keep real data in every mix.