Why it matters

ZeRO-Offload enables single-GPU training of big models. Understanding shapes design.

Advertisement

The architecture

Optim states in CPU.

Forward + backward on GPU.

Update on CPU.

ZeRO-Offload flowParams on GPUshardOptim + grads on CPUlarge RAMUpdateon CPUCPU-GPU transfer bottleneck; NVMe further degrades but enables larger
ZeRO-Offload.
Advertisement

How it works end to end

Optim states 4-8x model size.

Move to CPU: fits big models.

Cost: PCIe transfer.

NVMe offload extends further.