QLoRA — 4-bit Quantized Base + LoRA for Consumer-GPU Fine-Tuning

Why it matters

QLoRA democratized large-model fine-tuning. Before it, only labs with H100 clusters could fine-tune 70B models. Now anyone with a gaming GPU can. Understanding it opens up experimentation.

The architecture

Base model: quantize to NF4 (4-bit NormalFloat, custom format for weight distribution).

LoRA adapters: FP16, trainable. Gradients flow through dequantization.

How it works end to end

NF4: information-theoretically optimal 4-bit format for normally-distributed weights (weights approximate this).

Double quantization: quantize the quantization constants themselves. Extra memory savings.

Paged optimizers: swap optimizer state to CPU when GPU memory tight.

Fine-tuning 70B on 24GB VRAM feasible. Slower than FP16 but works.