Why it matters
Adafactor cuts optimizer memory. Understanding shapes large-model training.
Advertisement
The architecture
Instead of full v for each param, store row + column stats.
Reconstruct approximation.
Advertisement
How it works end to end
Row + col sum of squares.
v_ij ~= row_i * col_j / total.
Optional momentum.
Relative step: scale-invariant.