Why architecture matters here
SLM distillation matters because it lets you concentrate capability into a smaller footprint. A 3B parameter model that has been distilled from a 70B teacher can approach the teacher's quality on the tasks you care about, at a fraction of the inference cost. On-device viability follows directly.
Cost is real and dramatic. Training a small model from scratch to reach a target quality takes tens of millions of tokens; distilling from a strong teacher takes hundreds of thousands. Distillation is the biggest lever in small-model training economics.
Reliability comes from careful pipeline design. Naive distillation produces students that echo teacher styles but hallucinate on edge cases; filtered, diverse, reasoning-rich distillation produces students that generalize.
The architecture: every stage explained
Walk the diagram top to bottom.
Teacher Model. A large, high-quality model. GPT-4, Claude Opus, Llama 3 405B, or an in-house model. The upper bound on student quality.
Task Data. Prompts that cover the tasks the student needs to handle. Curated for diversity and domain coverage.
Student Model. A smaller model. 1B to 8B parameters typically. Starts from a pretrained base (Llama, Mistral, Qwen small).
Response Distillation. Simplest form. Teacher generates completions for prompts; student trains via SFT on (prompt, teacher_response) pairs. Works well for many tasks.
Logit Distillation. Deeper form. Match teacher's per-token probability distribution, not just the top token. Requires access to teacher logits (own model or open weights). Preserves more nuance.
Reasoning Distillation. Teacher generates full chain-of-thought reasoning; student learns to reason similarly. Critical for math, code, and multi-step tasks. Wei et al. 2022 and DeepSeek R1 popularized.
Diverse Sampling. Run the teacher multiple times per prompt with different temperatures; keep varied outputs to expose the student to multiple valid responses.
Filtering. Not every teacher response is good. Filter by task-specific quality checks (correct math, executable code) or by teacher self-critique.
Evaluation. Task-specific benchmarks plus general capability benchmarks (MMLU, HumanEval, GSM8K). Measure how close student comes to teacher.
Deployment. Quantize the distilled student (INT4 usually) for edge deployment. Serve via llama.cpp, MLC, or Core ML.
End-to-end distillation flow
Trace a distillation. You want a 3B model that handles customer-support queries well, replacing an expensive GPT-4-based service.
Curate 100k support queries from your history + synthetic diversifications. Send each to GPT-4; collect responses.
Add reasoning: for each query, prompt GPT-4 with "think step by step then answer." Capture the full reasoning trace.
Filter: keep only responses where GPT-4's self-critique rated ≥ 4/5 on helpfulness. Remove ~15% low-quality.
Train the student: fine-tune Llama-3.2-3B on the (prompt, reasoning, answer) tuples using SFT. Learn to produce both reasoning and answer.
Evaluate on held-out queries. Human eval says: student matches teacher 87% of the time; teacher 6% of the time. 7% ties. Good enough for the use case.
Quantize student to INT4. Deploy on edge or cheap cloud GPUs.
Cost per query: teacher $0.02; student $0.002. 10x reduction. Quality delta: within acceptable band. Ship.