Core concept
2x memory + 2x throughput on tensor cores vs FP32. BF16 wider range than FP16 (no loss scaling).
Advertisement
How it works
Some ops need FP32 for stability. Norms, softmax, reductions.
Advertisement
Trade-offs + gotchas
All modern training. FP8 for latest models with Transformer Engine.