BF16: 8 exp bits (FP32 range), 7 mantissa. FP16: 5 exp bits (narrow), 10 mantissa.

What you're seeing

FP16 range: ±65504. Easily overflows in attention. BF16 has FP32's range. Slightly less precise but training is robust to it.

★ KEY TAKEAWAY
BF16 = FP32 range + half mantissa. Faster, half memory, training-safe. The right choice for modern CPU/GPU training.
▶ WHAT TO TRY
  • Compare the bit layouts visually.
  • FP16 has only 5 exponent bits — overflows easily in attention. BF16 has FP32's 8 — won't.
← Back to Transformer Math & CPU SLM