Why it matters

Optim state often dominates memory. Understanding shapes optimizer choice.

Advertisement

The architecture

AdamW: m (fp32) + v (fp32) = 8 bytes.

Lion: 1 momentum = 4 bytes.

Adafactor: row + col stats.

Optimizer state memoryAdamW8 bytes/paramLion4 bytes/paramAdafactorsub-linearFor 70B: AdamW = 560GB optim state alone; explains ZeRO-1 popularity
Optim state.
Advertisement

How it works end to end

Master weights fp32 + optim state: ~12 bytes/param.

Total optim mem huge for big models.

ZeRO-1: shard just optim state.