Why it matters

Mixed precision is baseline modern training. Understanding shapes design.

Advertisement

The architecture

fp16 or bf16 forward + backward.

fp32 master copy of weights.

Loss scaling (fp16 only).

Mixed precisionLow-prec computefp16/bf16fp32 master weightsfor accum + updateLoss scale (fp16)avoid underflowPyTorch autocast + GradScaler automate; bf16 typically simpler than fp16
Mixed precision.
Advertisement

How it works end to end

PyTorch: torch.cuda.amp autocast.

GradScaler for fp16 loss scaling.

bf16: no loss scaling.

Speed: 2-3x over fp32.