Why it matters
DoRA improves LoRA quality. Understanding shapes advanced PEFT.
Advertisement
The architecture
W = ||W|| * (W / ||W||).
LoRA on direction W / ||W||.
Separately train magnitude vector.
Advertisement
How it works end to end
Liu et al. 2024.
Small overhead per column magnitude.
PEFT library supports.