Math
W' = W + BA where W is d×d (frozen), B is d×r, A is r×d. Trainable params: 2·d·r. For d=4096, r=8: 65k vs 16M.
Advertisement
Which weights
Attention: Q + V typical. Sometimes all Q/K/V/O + MLPs. More adapted layers → more expressivity + params.
Advertisement
Rank choice
r=8 default. r=16, 32, 64 for harder tasks. Higher r = more params = risk overfit. Often r=8 enough.