Why it matters

Distillation enables efficient small models. Understanding shapes SLM training.

Advertisement

The architecture

Loss: KL divergence between student + teacher distributions.

Optionally: hard-label CE too.

Distillation flowTeachersoftmax outputsStudentmatches distributionKL lossdistributional matchTemperature > 1 softens both distributions; better gradient signal
Distillation.
Advertisement

How it works end to end

Temperature T: softmax(logits / T).

KL(student || teacher).

Alpha: mix hard + soft loss.

DistilBERT, TinyLlama used this.