Why it matters

ZeRO-1 is first sharding. Understanding shapes ZeRO.

Advertisement

The architecture

Optim state sharded across DP replicas.

Each device owns 1/N optim state.

Grad reduction + optim update on owner.

ZeRO Stage 1 mathOptim state shard1/N per deviceGrad reduce-scatterto owner deviceUpdate + all-gatherparams after step4x memory reduction; combined with DP; foundation of DeepSpeed
ZeRO Stage 1 math.
Advertisement

How it works end to end

Rajbhandari 2020 ZeRO.

Simplest ZeRO stage.

Communication similar to DDP.