Why it matters

BF16 replaced FP16 as the training precision of choice. Understanding why explains modern LLM training practices.

Advertisement

The architecture

Format: 1 sign, 8 exponent (same as FP32), 7 mantissa.

Range: same as FP32 (~10^-38 to 10^38).

BF16 propertiesSame range as FP32no underflowLess mantissaless precisionHardware supportedAmpere+Preferred for training over FP16; no loss scaling needed
BF16 vs FP16.
Advertisement

How it works end to end

Training: no loss scaling needed. Gradients don't underflow.

Precision: 7 mantissa bits (~3 decimal digits). Less than FP16's 10 (~3-4 decimal digits) — wait, similar in decimal terms but FP16 has 1024 vs BF16 has 128 mantissa distinct values in same exponent range.

Hardware: A100, H100, TPU support BF16 natively. Older hardware falls back to FP32.