Why it matters

LoRA is standard fine-tuning. Understanding shapes strategy.

Advertisement

The architecture

Freeze W.

Train A, B: rank-r update.

Merge: W_new = W + BA.

LoRA updateFrozen Wd x dLow-rankBA where r << dCombinedW + BARank 8-64 typical; adapts <1% of params
LoRA structure.
Advertisement

How it works end to end

Init: A random, B zero (so BA = 0 initially).

Merge at inference for no overhead.

Alpha: LoRA scaling factor.