Why it matters

ZeRO enabled the scaling of modern LLMs. Understanding shapes memory-optimized training.

Advertisement

The architecture

Stage 1: optimizer state (moments) sharded.

Stage 2: + gradients.

Stage 3: + weights.

ZeRO stagesZeRO-1optimizer onlyZeRO-2+ gradientsZeRO-3+ weights (full)Memory reduction linear with GPU count; communication grows with sharding
ZeRO progression.
Advertisement

How it works end to end

Memory savings: proportional to sharding. Stage 3 gives N-fold reduction.

Communication cost: more sharding = more all-gather / reduce-scatter.

Trade-off: memory vs communication.