Why it matters

DoRA improves LoRA. Understanding shapes fine-tuning.

Advertisement

The architecture

W = ||W|| * (W / ||W||).

LoRA on direction.

Learn magnitude separately.

DoRA structureDecompose Wmagnitude + directionLoRA on directionrank-r updateLearn magnitudecolumn-wiseConsistent quality gains over vanilla LoRA in benchmarks
DoRA math.
Advertisement

How it works end to end

Directional LoRA + magnitude vector.

Small param increase over LoRA.

Better quality: closes gap to full fine-tune.