Why it matters
Distillation lets you compress a 70B model's capability into 7B parameters. That's an order of magnitude in inference cost. Understanding it is essential for anyone shipping SLMs.
The architecture
Teacher: strong pretrained model. Student: smaller model, initialized randomly or from smaller pretrained.
Data: prompts. Teacher generates soft outputs (probability distributions over tokens). Student trained to match.
How it works end to end
Loss: KL divergence between student and teacher output distributions. Temperature scaling smooths distributions for better learning.
Variants: response-level (match final output), feature-level (match intermediate representations), attention distillation, self-distillation (student = teacher, iterative refinement).
Data quality: synthetic data from teacher often works better than human data. Focus on tasks teacher does well.