Why it matters

ZeRO enables large-model training. Understanding shapes memory optimization.

Advertisement

The architecture

Stage 1: shard optim states.

Stage 2: shard gradients too.

Stage 3: shard params too.

ZeRO stagesStage 1shard optimStage 2+ shard gradsStage 3+ shard paramsStage 3 = FSDP; Stage 2 balances memory + comm for many workloads
ZeRO.
Advertisement

How it works end to end

Stage 1: 4x memory reduction.

Stage 2: 8x.

Stage 3: linear in N GPUs.

Comm cost rises with stage.