Why it matters

ZeRO-2 further reduces memory. Understanding shapes ZeRO.

Advertisement

The architecture

Grads reduced + kept only on owner.

Optim state on owner.

Params replicated still.

ZeRO Stage 2 mathGrads sharded1/N per deviceOptim state sharded1/N per deviceParams replicatedstill on all devices8x memory reduction over baseline DP; comm similar to Stage 1
ZeRO Stage 2 math.
Advertisement

How it works end to end

DeepSpeed + FSDP.

Params still full.

Comm reduce-scatter grads.