Self-Instruct
Wang et al 2022. Seed instructions + LLM generates more. Alpaca dataset created this way. Diversity + quality trade-offs.
Advertisement
Distillation
Teacher = big model. Student = small model. Train on teacher outputs. Vicuna, Phi trained this way.
Advertisement
Evol-Instruct
Iteratively make instructions harder ('rewrite this task with more constraints'). WizardLM approach.