Loss Scaling for fp16 Training

Why it matters

Loss scaling critical for fp16 training. Understanding shapes precision management.

The architecture

Loss = actual_loss * S.

Backprop: gradients * S.

Unscale before optimizer.

How it works end to end

Fixed vs dynamic scaling.

Dynamic: increase S if no overflow; halve if overflow.

Check for inf/nan.

bf16 doesn't need loss scaling.