SLM Fine-Tuning — Adapting Small Models to Specific Tasks

Why it matters

Task-specific SLMs are the sweet spot for many production applications: cheaper than GPT-4, better than generic SLMs on the target task. Fine-tuning is how you get there.

The architecture

Data: 100-10000 high-quality examples typical. Small dataset can fine-tune; more data = better generalization.

Method: LoRA or QLoRA for efficiency; full fine-tune when you have compute.

How it works end to end

Learning rate: 1e-4 to 5e-5 for LoRA. Full fine-tune uses 1e-5.

Epochs: 1-3 typical. More causes overfitting on small datasets.

Data mixing: mix task data with some generic instruction data to prevent capability loss.

Evaluation: measure on target task + broad capabilities. Fine-tuning can silently damage other skills.