Why architecture matters here
SLMs fail when teams distill without a target task or without the right data. A 7B distilled from a 70B on generic web text ends up mediocre everywhere. A 7B distilled with a curated task-specific dataset can match or beat the 70B on that task.
The architecture matters because each step has a design decision: teacher choice (bigger is not always better), dataset curation (task-specific beats generic), loss formulation (KL vs CE mix), temperature (how soft to make targets), and evaluation (task-specific benchmarks that reflect the product surface).
Getting the pipeline right yields models that ship and delight users at cost that scales.
The architecture: every piece explained
The top strip is the training loop. Teacher model is the large model whose behavior you want to compress. Dataset curation gathers task-representative prompts + expected outputs, augmented with synthetic teacher-generated examples. Teacher forward produces soft targets — full logit distributions — that carry more information than hard labels alone. Student model is the small architecture; often a purpose-designed compact model.
The middle row is the loss formulation. KD loss combines KL divergence between student and teacher distributions with a cross-entropy loss on hard labels; the mix is tuned. Temperature softens both distributions — higher T gives smoother targets that teach relative preferences. Curriculum orders examples from easy to hard so the student builds capability gradually. Data augmentation uses the teacher to generate additional supervised examples where labeled data is scarce.
The lower rows are practical delivery. Eval harness measures per-task metrics, latency, and cost against the teacher. Deployment target — edge, CPU, small GPU — drives quantization and architecture choices. Ops handles versioning, A/B testing, rollback, and feedback labels that flow back into the next distillation cycle.
End-to-end flow
End-to-end: a team distills a 70B teacher into a 3B student for on-device chat summarization. They curate 500k domain-specific summarization examples and augment with 2M teacher-generated pairs. Teacher forward produces soft logit distributions at temperature 4. Student trains for 3 epochs with KD loss = 0.7 KL + 0.3 CE. Curriculum starts with short passages, grows to long documents. Eval: ROUGE-L 0.42 vs teacher 0.45 — 6% gap at 1/20 the size. Latency on device: 40 tokens/s vs 4 for the teacher. Ship. A/B against the previous 3B baseline shows +3% user acceptance. Rollback flag ready. User feedback logs feed the next distillation.