Why it matters

BF16 dominates modern training. Understanding shapes precision choice.

Advertisement

The architecture

Same exponent as fp32 (8 bits).

Mantissa: 7 bits.

Range: like fp32; precision: coarser.

BF16BF16 format1 sign + 8 exp + 7 mantissafp32 rangeno overflowReduced precisionless than fp16 mantissaTPUs pioneered; Ampere+ and Hopper full-throughput BF16
BF16.
Advertisement

How it works end to end

Introduced by Google TPU.

NVIDIA A100+ full-rate.

Master weights + FSDP shard.

No loss scaling needed (vs fp16).