LoRA — Low-Rank Adaptation for Efficient Fine-Tuning

Why it matters

LoRA changed fine-tuning economics. Where full fine-tuning of a 70B model needs multiple 80GB GPUs, LoRA fits on a single consumer GPU. This has democratized fine-tuning.

The architecture

Weight matrix W (d × d). Learn ΔW = A × B where A is (d × r), B is (r × d), r << d.

Final weight during training: W + ΔW. Inference: merge or keep separate.

How it works end to end

Which layers: attention Q, K, V projections most impactful; MLP layers optional. LoRA on all linears typical for quality; on Q, V only for max efficiency.

Rank r: 8-64. Higher r = more capacity = closer to full fine-tune quality.

QLoRA: quantize base model to 4-bit, LoRA in FP16 on top. Even more efficient, minimal quality loss.

Adapter swap: multiple LoRA adapters for different tasks; swap without reloading base.