Why it matters

FSDP is standard for large models. Understanding shapes design.

Advertisement

The architecture

Shards params across GPUs.

All-gather before forward/backward.

Reduce-scatter grads.

FSDP flowSharded params1/N per GPUAll-gatherbefore forwardReduce-scattergrads shardSimilar to ZeRO Stage 3; PyTorch FSDP + FSDP2 improve overlap
FSDP.
Advertisement

How it works end to end

Shard: params + grads + optim state (like ZeRO-3).

Communication increases 1.5x DDP.

Enables larger models.