Why architecture matters here
Fine-tuning architecture matters because method choice shapes cost by orders of magnitude. Full fine-tuning of a 70B model requires 100+ H100s for hours; QLoRA on the same model runs on a single H100 in hours. Task quality difference is often small; wasted money is often large.
Cost is the primary lever. Match method to actual quality need.
Reliability of the fine-tuned model depends on evaluation. Task metrics catch specific improvements; capability benchmarks catch regressions on general skills; safety evals catch new harm modes.
The architecture: every method explained
Walk the diagram top to bottom.
Base Model. Pretrained + safety-tuned base — Llama 3, Mistral, Qwen. Start here.
Task Dataset. Supervised fine-tuning (SFT) pairs or preference pairs (DPO). Quality > quantity.
Method Choice. LoRA for standard adaptation; QLoRA when budget-constrained; full FT for maximum quality (rarely necessary); DPO for preference alignment.
LoRA. Rank-r matrix additions on attention/MLP projections. r=8-64 typical. Trainable params: 0.1-1% of base. Fast, cheap, mergeable.
QLoRA. Base quantized to NF4 (4-bit); LoRA adapters trained in higher precision. Fine-tune 70B on single 80GB GPU.
Full FT. All params trainable. Best quality per token but expensive. Also more prone to catastrophic forgetting.
DPO / KTO. Preference optimization. DPO uses chosen/rejected pairs; KTO uses per-example thumbs up/down. Simpler than PPO-RLHF.
Merge Adapter. LoRA adapters merge into base weights via low-rank multiply-add. Deploy as one model, no adapter overhead.
Evaluation. Task-specific benchmark; general capability suite (MMLU, HumanEval, GSM8K); safety (jailbreak resistance, harm categories).
Serving. Serve merged model or serve base + attach adapter at load. Adapter-based serving lets you A/B multiple variants on one base.
End-to-end fine-tune flow
Trace a fine-tune. Team wants to specialize a general model for customer support responses.
Curate 10k support conversations with high-quality responses. Format as SFT pairs (input + target response).
Choose QLoRA: fits budget (single GPU), quality sufficient. Rank r=32. Target modules: q_proj, k_proj, v_proj, o_proj.
Train via HuggingFace TRL or Axolotl. 3 epochs; ~4 hours on a single H100. Adapters saved: 250MB.
Merge adapter into base (or serve separately for A/B). Eval:
- Task benchmark: customer support satisfaction judged by GPT-4o goes from 3.8 to 4.4.
- General capability: MMLU drops 0.2 pts (acceptable). HumanEval unchanged.
- Safety: jailbreak resistance holds; PII avoidance improved.
Canary deploy: 5% of support tickets to new model. Metrics: resolution rate up, CSAT up. Full deploy.
Iterate: two months later, retrain with new tickets. Same pipeline; rebuild adapter.