Why architecture matters here
Distillation earns its place in the architecture because the alternatives are worse. Running the teacher in production solves quality but not cost or latency. Training a small model from scratch on hand-labeled data solves cost but usually not quality — small models trained on limited hard labels underperform, because the label alone is a thin signal and good labeled data is expensive to produce at scale. Distillation threads the needle: the teacher becomes a cheap, tireless labeler that produces not just answers but rich, graded supervision on as much unlabeled data as you can feed it, and the student learns a compact function that captures the teacher's behavior on the task's actual distribution. You are, in effect, compressing the part of the teacher's capability that this task uses into a model sized for exactly that.
The reason soft targets matter so much is information density. A hard label collapses the teacher's nuanced view into a single class; the soft distribution preserves the similarity structure — that a lynx is more cat-like than car-like, that this sentiment is 70% positive and 30% mixed rather than flatly positive. This inter-class information is a regularizer and a curriculum in one: it tells the student where the decision boundaries are soft and where they are sharp, which is precisely the knowledge that lets a small model generalize from limited data. For language and reasoning tasks, distilling the teacher's step-by-step rationale extends the effect — the student learns the intermediate structure, not just the endpoints, and a small model that has seen the teacher's reasoning can solve problems a same-size model trained on answers alone cannot.
The central architectural constraint is that the student inherits only what it is shown. Distillation transfers the teacher's behavior on the data distribution you distill over — and nothing else. A student distilled on common cases will be excellent on common cases and blind on the rare ones the teacher would have handled, because it never saw them. This is why data curation, not the loss function, is usually the difference between a student that replaces the teacher and one that replaces it only on the demos. The distribution you distill over is the capability you get.
The architecture: every piece explained
Top row: generating supervision. The teacher model — large and accurate — is run over a corpus of unlabeled data: real or synthetic prompts and inputs drawn from the target task's distribution. For each input the teacher produces soft targets: for a classifier, the full logits or probability vector; for a generative model, the token-level distributions or sampled outputs. The student model — small and fast — is the artifact being trained to reproduce those targets. The teacher runs once to generate the dataset; from then on training is cheap.
Middle row: the transfer mechanism. The distillation loss trains the student to match the teacher's distribution, typically a KL-divergence between student and teacher probabilities, often combined with a standard task loss on any hard labels available — the weighting balances 'mimic the teacher' against 'be correct'. Temperature is the knob that makes soft targets useful: dividing the logits by a temperature above 1 softens the distribution, amplifying the small probabilities that carry the inter-class 'dark knowledge' so the student actually learns from them rather than from the argmax alone. For reasoning tasks, rationale or chain-of-thought distillation has the teacher emit its reasoning trace and trains the student to produce it, transferring process as well as outcome. The eval gate compares the trained student against the teacher on a held-out set: only a student that clears the quality bar for the task proceeds.
Bottom rows: the surrounding discipline. Data curation is the load-bearing step — the distillation corpus must cover the domain, including the hard, rare, and adversarial cases, because the student's competence is exactly the distribution it was distilled over; curation is where you decide what capability the student will have. Once the student passes the gate it is deployed to serve production traffic at a fraction of the teacher's cost and latency. The ops strip keeps it honest over time: drift monitoring (is production traffic moving away from the distilled distribution?), scheduled re-distillation as the teacher improves or the domain shifts, teacher versioning so you know which teacher a student came from, and regression evals so a new student never silently loses a capability the old one had.
End-to-end flow
Follow a concrete project: a company runs a frontier LLM to triage millions of support tickets a day — classify intent, extract entities, and draft a first response — and the bill is dominated by that per-ticket call. They decide to distill a small student to handle the bulk of tickets, keeping the teacher only for the hard tail.
Curation comes first and takes the most care. They assemble a distillation corpus from a year of real tickets, deliberately oversampling the rare intents, the multilingual cases, and the angry-escalation patterns that matter most — because they know the student will only be as good as this coverage. They run the teacher over the whole corpus, capturing for each ticket the teacher's intent distribution (soft targets), its extracted entities, and its chain-of-thought rationale for the triage decision. This generation pass is a one-time cost paid at batch rates.
Training the student uses a temperature of around 2 to soften the intent distributions, a KL loss to match the teacher plus a cross-entropy term on the cases where a human label exists, and rationale distillation so the student learns to produce a brief reasoning trace before its classification — which, as expected, lifts accuracy on the ambiguous tickets where the reasoning matters. The student is perhaps a fortieth of the teacher's size. At the eval gate they measure it not on average accuracy alone but per-intent, and they find the classic distillation result: the student matches the teacher within a point or two on common intents but lags on two rare intents that were under-represented even after oversampling. Rather than ship a student blind on those, they route: the student handles the intents where it cleared the bar, and tickets it classifies with low confidence — or that fall into the weak intents — escalate to the teacher. The result is that the great majority of tickets are served by the cheap student and only the hard tail reaches the expensive teacher, cutting cost enormously while holding quality.
Operations close the loop. Months later, drift monitoring shows a new product line generating tickets the student handles poorly — a distribution the original corpus never contained. This is the expected lifecycle, not a surprise: they re-curate with the new tickets, re-run the (now upgraded) teacher, re-distill, and gate the new student against the old one with regression evals to guarantee it lost no prior capability while gaining the new one. Teacher versioning records exactly which teacher produced which student, so a regression can always be traced to its source. The distilled student is not a one-time artifact but a continuously refreshed compression of the teacher's behavior on a moving target.