Why it matters

FSDP is native large-model training in PyTorch. Understanding shapes training on standard PyTorch.

Advertisement

The architecture

Wrap model with FSDP.

Weights, gradients, optimizer state sharded across data-parallel ranks.

Gather + shard around forward/backward.

FSDP mechanicsWeights shardedeach rank holds partGather before computetemporarily fullBackward + shardrelease memoryFSDP2 (Composable FSDP): more flexible, better perf; use for new code
FSDP flow.
Advertisement

How it works end to end

All-gather: assemble full parameter shard before compute.

Reduce-scatter: shard gradients after backward.

Activation checkpointing: recompute activations to save memory.

FSDP2: newer API with better ergonomics + performance.