Why it matters

Optim state offload frees GPU memory. Understanding shapes training.

Advertisement

The architecture

Adam m + v moments to CPU.

Gradient update on CPU.

Result copied to GPU.

Optim state offload flowOptim state8 bytes/paramMove to CPUmoments + master weightsUpdate + copy backPCIe transferZeRO-Offload standard; 100B+ models on modest GPU cluster possible
Optim state offload math.
Advertisement

How it works end to end

Adam 8 bytes/param.

Master weights (fp32) too.

Update on CPU.

ZeRO-1 offload variant.