Why it matters
DDP is baseline multi-GPU. Understanding shapes design.
Advertisement
The architecture
Each GPU has full model copy.
Batch split across GPUs.
Grads all-reduced.
Optimizer step per replica.
Advertisement
How it works end to end
All-reduce: ring or tree.
Overlap comm with backward.
Effective batch = per-GPU batch * world size.