SLM distillation from LLM architecture — teacher, student, rationale

Why architecture matters here

Distillation fails when task distribution is narrow or teacher rationales are wrong. Architecture matters because prompts + rationale + KD + eval compose.

The architecture: every piece explained

The top strip is teacher. Teacher LLM. Prompt set. Teacher outputs. Student model.

The middle row is training. SFT on rationales. KD loss. RLHF/DPO. Eval vs teacher.

The lower rows are ops. Task distribution. Metrics. Ops — teacher cost + iteration + serving.

End-to-end flow

End-to-end: broad prompt set (100k) covering intended tasks. Teacher generates answer + CoT for each. Student SFT on rationale pairs. KD loss adds soft distributions. DPO from preference pairs. Eval: student vs teacher on held-out.