Memory savings
4-bit vs 16-bit base = 4x reduction. Adapter minimal. 65B fine-tune on single 48GB GPU. 7B on 12GB.
Advertisement
NF4 quantization
Normal Float 4: quantile-based scheme optimized for normal-distributed weights. Better than uniform 4-bit.
Advertisement
Double quantization
Also quantize quantization constants. Small additional saving. Standard in QLoRA.