Core concept
ZeRO-Offload: offload states to CPU. ZeRO-Infinity: offload to NVMe.
Advertisement
How it works
DeepSpeed complex config. FSDP native to PyTorch increasingly preferred.
Advertisement
Trade-offs + gotchas
Historical importance. Still used in Megatron-DeepSpeed for very large models.