Why it matters

ZeRO-3 is FSDP foundation. Understanding shapes frontier training.

Advertisement

The architecture

Params sharded across DP.

All-gather before forward + backward.

Reduce-scatter grads.

Only 1/N of everything per device.

ZeRO Stage 3 mathParams sharded1/N per deviceAll-gather forwardbefore computeReduce-scatter grads1/N after backward= FSDP; frontier LLM training; 1.5x DDP communication trade-off
ZeRO Stage 3 math.
Advertisement

How it works end to end

FSDP = ZeRO Stage 3.

Comm 1.5x DDP.

Linear memory in DP degree.

Frontier training.