Why it matters

Gradient memory doubles param memory. Understanding shapes reduction techniques.

Advertisement

The architecture

Grads: 2 bytes (bf16) or 4 (fp32) per param.

Or shard across GPUs.

Gradient memory optionsGrad tensor2-4 bytes/paramShard (ZeRO-2)1/N per GPUOr bf16 gradhalf fp32ZeRO-2 shards grads across GPUs; FSDP is Stage 3 with param shard too
Gradient memory.
Advertisement

How it works end to end

bf16 grads: 2 bytes.

FP32 grads: 4 bytes.

ZeRO-2: 1/N grads per GPU.

ZeRO-3 / FSDP: 1/N params too.