Why it matters

FP16 was the first big win for ML precision reduction. Understanding it clarifies why BF16 arose and how mixed precision training works.

Advertisement

The architecture

Format: 1 sign bit, 5 exponent bits, 10 mantissa bits.

Range: roughly ±65504. Precision: much less than FP32.

FP16 characteristicsHalf memory2 bytes / valueFaster computeon tensor coresLimited range±65504Tensor cores accelerate FP16 matmul; mixed precision keeps loss scaling
FP16 properties.
Advertisement

How it works end to end

Dynamic range: narrower than FP32. Gradients can underflow.

Loss scaling: multiply loss by large factor before backprop to keep gradients in range. Standard mixed precision technique.

Hardware: tensor cores support FP16 with dramatic speedup vs FP32 (up to 8x).

Alternative: BF16 has wider range, less mantissa. Often preferable for training.