Why architecture matters here
Why does this need real architecture? Because every failure mode of synthetic data is silent and delayed. A bug in a serving system pages you tonight; a contaminated or degenerate training batch shows up weeks later as an eval score that is either fraudulently high (your test questions leaked into training) or mysteriously flat (a million near-duplicate examples taught the model nothing new). By the time you notice, the batch has been mixed, shuffled, and trained on, and without lineage you cannot even identify which examples to pull. The pipeline exists to make these invisible failures visible at the stage where they are still cheap to fix.
The economics force the funnel shape. Generation costs real money per example, but filtering costs less the earlier it runs: a regex that rejects malformed outputs costs microseconds, an embedding dedup costs a vector lookup, an LLM judge costs a model call, and a full training run to discover the batch was bad costs thousands of dollars and a week. So the architecture is a strict cost-ordered funnel — cheap filters first, expensive judgment later, training-based validation last and rarest. And because the generator is itself a model with its own biases, diversity must be engineered in at the prompt stage — personas, topics, difficulty knobs, seed examples — rather than hoped for from temperature. Left to defaults, a generator produces a thousand variations of the same five examples, and no downstream filter can add information that generation never created.
The architecture: every piece explained
The seed corpus anchors everything: a few hundred to a few thousand real, vetted examples plus a taxonomy of what coverage means — domains, skills, difficulty tiers, languages. The prompt templater crosses that taxonomy into generation prompts (persona × topic × difficulty × format), each carrying a handful of seed examples as style anchors; this grid, not sampling temperature, is where diversity comes from. The generator is typically the strongest model you can afford, run wide; a self-critique pass immediately asks the same or a second model to verify, rewrite, or reject its own output — cheap, and it lifts yield noticeably before anything downstream runs.
Then the filter stack, cheapest first. Heuristic filters: length bounds, format validation, answer-parses-and-executes checks for code and math. Dedup in two passes — exact and MinHash for near-duplicates, then embedding-space clustering that caps how many examples may occupy one semantic neighborhood, which is the actual defense against mode collapse. The judge panel scores survivors on correctness, faithfulness to the instruction, and safety — multiple judge prompts or models, because a single judge has exploitable blind spots and drifts. Decontamination runs both n-gram overlap and embedding similarity against every evaluation set you care about, and it is a hard gate, not a score. The mixture composer assembles the final training blend — synthetic-to-real ratios set per domain, not globally. And the lineage store stamps every surviving example with its template, seeds, generator model and version, judge scores, and batch ID — the provenance record that makes recalls possible. The eval harness closes the loop: no batch ships until a model trained (or LoRA-tuned) on it beats the baseline on held-out evals.
End-to-end flow
Walk one batch through. The goal: 50k examples of multi-step SQL reasoning. The templater crosses 40 personas, 200 schema topics, and 4 difficulty tiers into 32k prompts, each with three seed examples; the generator produces four candidates per prompt — 128k raw. Self-critique immediately rejects 22k (wrong answer, refused, format drift). Heuristics — does the SQL parse, does it run against the schema, does the answer match execution — cut 18k more. Dedup is the surprise: MinHash finds 9k near-copies, and embedding clustering caps another 12k that all cluster around the same three JOIN patterns the generator loves. 67k reach the judges.
The judge panel — two models, three rubrics — scores each survivor; the threshold keeps 54k. Decontamination then removes 210 examples with 8-gram overlap against the team's SQL eval suite and 90 more by embedding similarity — a 0.5% hit rate that would have inflated the eval by several points. The mixture composer blends the surviving 53k at a 30:70 synthetic-to-real ratio for this domain, the lineage store stamps every row, and the eval harness fine-tunes a candidate: +4.1 on the SQL suite, no regression on general benchmarks. The batch ships — and every example in it can be traced back to its template and generator version. Total yield: 41% of raw generation, which is healthy; a pipeline yielding 90% is not efficient, it is under-filtered.