Mixed Precision Training — BF16 on CPU

Anatomy of a floating-point number

Every IEEE-style float is three fields packed into a fixed number of bits: a sign bit, an exponent field, and a mantissa (also called the significand or fraction). The value is reconstructed as (-1)^sign × 1.mantissa × 2^(exp - bias). The mantissa is the significant digits; the exponent slides the binary point to position those digits anywhere from the very small to the very large.

This split is the key to everything that follows. The exponent width sets the range — how big and how small a number you can represent before you overflow to infinity or underflow to zero. The mantissa width sets the precision — how finely you can distinguish two nearby numbers, i.e. the size of the gap (the ulp, unit in the last place) between one representable value and the next. Critically, precision is relative: near 1.0 a format with an m-bit mantissa has a gap of 2^-m, and near 1000 the gap is 2^-m × 1024. A float is a ruler whose tick spacing grows with the magnitude of what you are measuring. When you shrink a 32-bit float to 16 bits, you must decide which of those two budgets — range or precision — to cut. That decision is the entire story of FP16 versus BF16.

Advertisement

FP32: the full-precision baseline

FP32 (IEEE single precision) is the reference everyone else is measured against: 1 sign bit, 8 exponent bits, 23 mantissa bits. The 8-bit exponent gives a dynamic range of roughly ±1.2e-38 up to ±3.4e38, and the 23-bit mantissa (24 bits of effective significand with the implicit leading 1) gives about 7 decimal digits of precision — a relative gap near 1.0 of 2^-23 ≈ 1.2e-7.

FP32  [ S | EEEEEEEE | MMMMMMMMMMMMMMMMMMMMMMM ]   32 bits
        1     8              23
        range  ~ +/- 3.4e38     precision ~ 7 decimal digits

For decades this was simply ‘what training runs in.’ It has enough range that activations, weights, and gradients essentially never overflow, and enough precision that the tiny weight updates of gradient descent actually stick. The problem is cost: every FP32 tensor is 4 bytes per element, and every FP32 matmul moves and multiplies those 4-byte values. Since modern training and inference are dominated by moving numbers through memory and multiply-accumulate units, FP32 is leaving a large factor of speed and memory on the table — if only you could do the heavy arithmetic in half the bits without breaking convergence. That ‘if only’ is what mixed precision delivers.

Advertisement

FP16: keep the precision, lose the range

FP16 (IEEE half precision) is the intuitive way to halve a float: 1 sign bit, 5 exponent bits, 10 mantissa bits. It spends its scarce bits on the mantissa, so near 1.0 it still resolves about 3–4 decimal digits (a gap of 2^-10 ≈ 9.8e-4). What it sacrifices is range: the 5-bit exponent caps the largest normal value at 65504 and puts the smallest normal number at about 6.1e-5 (2^-14), with subnormals limping down to about 6e-8 (2^-24) before flushing to zero.

FP16  [ S | EEEEE | MMMMMMMMMM ]   16 bits
        1    5         10
        max normal   65504        smallest normal ~ 6.1e-5
        subnormals reach ~ 6e-8, then underflow to 0

That narrow range is the source of nearly every FP16 headache. Gradients in a deep network are routinely far below 6e-5 — especially early in training and in the lower layers — so a large fraction of them round to exactly zero, and zero gradients mean those parameters simply stop learning. On the high end, a few large activations or an unlucky attention score can punch through 65504 and produce an inf that poisons the whole backward pass. FP16 works, but only with a crutch — loss scaling — to drag the gradients back into representable range.

BF16: keep the range, lose the precision

BF16 (bfloat16, Google Brain’s format) makes the opposite cut: 1 sign bit, 8 exponent bits, 7 mantissa bits. The exponent is identical to FP32’s, so BF16 has the same enormous dynamic range — roughly ±3.4e38 — and essentially never overflows or underflows on values that FP32 would have handled. The price is precision: with only 7 mantissa bits, the gap near 1.0 is 2^-7 ≈ 7.8e-3, giving barely 2–3 decimal digits. BF16 is a coarse ruler with a very long reach.

BF16  [ S | EEEEEEEE | MMMMMMM ]   16 bits
        1     8           7
        range ~ +/- 3.4e38   (same as FP32)   precision ~ 2-3 digits

FP32  [ S | EEEEEEEE | MMMMMMMMMMMMMMMMMMMMMMM ]
BF16  [ S | EEEEEEEE | MMMMMMM ]  <-- just the top 16 bits of FP32

Notice the beautiful consequence of matching FP32’s exponent: a BF16 value is very nearly just the top 16 bits of the corresponding FP32 value. Converting FP32→BF16 is close to truncation, and BF16→FP32 is close to zero-padding. That near-trivial conversion, plus the fact that the range problems that force loss scaling in FP16 simply do not occur, is why BF16 has become the default low-precision training format — and, as we will see, why it is especially at home on a CPU.

Range versus precision: the trade in one table

Put the three formats side by side and the exponent/mantissa trade is stark. The exponent column governs whether your numbers survive at all; the mantissa column governs how accurately they are recorded.

Format	Sign	Exponent	Mantissa	Max value	Rel. precision
FP32	1	8	23	~3.4e38	~1.2e-7 (~7 digits)
FP16	1	5	10	65504	~9.8e-4 (~3-4 digits)
BF16	1	8	7	~3.4e38	~7.8e-3 (~2-3 digits)

Read across the rows and the philosophy of each format is obvious. FP16 and BF16 both weigh 16 bits, but FP16 buys three extra mantissa bits by giving up three exponent bits, and BF16 does the reverse. The deep-learning verdict, learned the hard way, is that range matters more than precision for training neural networks. Networks are remarkably tolerant of noisy, low-precision weights and activations — stochastic gradient descent is itself a noisy process, and rounding error just adds a little more — but they are not tolerant of gradients silently becoming zero or activations becoming infinity. BF16 protects the thing that matters (range) and spends the thing the network can absorb (precision). That is why, on hardware that supports it, BF16 generally trains more robustly and with less babysitting than FP16.

Why compute in low precision at all?

The motivation is entirely about memory and speed, and both come from the same fact: a 16-bit value is half the bytes of a 32-bit value. On the memory side, activations stored for the backward pass, the largest transient cost of training, halve. A transformer’s activation memory scales with batch × sequence × hidden × layers; cutting the per-element size from 4 bytes to 2 lets you fit roughly twice the batch or twice the context in the same footprint. Weights and gradients moving across the memory bus also halve.

On the speed side, most training and inference is bandwidth-bound: the arithmetic units sit idle waiting for numbers to arrive from memory. Halving the bytes per number roughly doubles the effective bandwidth and doubles how much of a weight matrix fits in cache. On top of that, dedicated low-precision matrix units — GPU tensor cores, and on CPUs the AVX-512 BF16 and AMX instructions — execute 16-bit multiply-accumulate at two to several times the throughput of FP32. The combined effect is commonly a 1.5–3× training speedup and a ~2× memory reduction for the low-precision tensors. The catch is that you cannot naively cast everything to 16 bits and hope for the best — some quantities are too delicate. Managing which stays high-precision is the ‘mixed’ in mixed precision.

The FP32 master copy of the weights

The first quantity too delicate to demote is the weights themselves — specifically, the copy that gets updated. Mixed precision keeps a full-precision FP32 master copy of every weight, and the optimizer applies its updates to that master copy. Each step, the master weights are cast down to BF16 (or FP16) for the forward and backward passes, but the authoritative state that persists across steps lives in FP32.

The reason is update swamping. Gradient-descent updates are typically minuscule relative to the weights they modify — a weight near 1.0 might receive an update of 1e-4 or smaller. If the weight is stored in 16 bits, that update can be smaller than a single ulp at the weight’s magnitude, so weight + update rounds right back to weight and the step is silently lost. Over thousands of steps those lost updates are the difference between learning and stalling. Keeping the master copy in FP32 — with its ~1.2e-7 resolution near 1.0 — means even tiny updates accumulate correctly. Note this bites BF16 harder than FP16, because BF16’s coarser 7-bit mantissa has an even larger ulp; the master copy is not an FP16-only patch, it is fundamental to both.

Update swamping: a worked example

Make the swamping concrete. Take a weight w = 1.0 and a per-step update Δw = -0.0001 (learning rate times gradient). The spacing of representable values near 1.0 — the ulp — is 2^-m for an m-bit mantissa:

ulp at 1.0:
  FP32  2^-23 ~ 1.2e-7      BF16  2^-7 ~ 7.8e-3      FP16  2^-10 ~ 9.8e-4

update  |dw| = 1e-4
  BF16:  1e-4  <  half-ulp (3.9e-3)   ->  1.0 - 1e-4  rounds to  1.0   (LOST)
  FP16:  1e-4  <  half-ulp (4.9e-4)   ->  1.0 - 1e-4  rounds to  1.0   (LOST)
  FP32:  1e-4  >> half-ulp (6e-8)     ->  1.0 - 1e-4  =  0.9999   (KEPT)

In both 16-bit formats the update is smaller than half the gap to the next representable number, so rounding-to-nearest snaps the result straight back to 1.0: the optimizer ‘stepped’ but nothing moved. Only the FP32 master copy records the change. Now run 10,000 steps of -1e-4 each: in FP32 the weight drifts to 0.0 as it should; in a 16-bit-only weight it might never leave 1.0, because each individual step keeps vanishing before it can accumulate. This is precisely why the master copy exists — it is the ledger with enough decimal places to record small deposits that the 16-bit working copy would round away.

Loss scaling: rescuing FP16 gradients from underflow

The FP32 master copy fixes the update side. FP16 has a second, separate problem on the gradient side: many gradient values are so small they fall below FP16’s minimum representable magnitude and underflow to zero before the optimizer ever sees them. Recall FP16’s smallest normal is ~6.1e-5 and it flushes to zero below ~6e-8. Activation gradients in real networks routinely live in the 1e-7 to 1e-10 region — a whole swath of them simply disappears.

The fix is loss scaling, and it is delightfully simple. Multiply the loss by a large constant S before calling backward. By the chain rule, every gradient in the network is then multiplied by the same S, shifting the entire gradient distribution up into FP16’s representable range where it survives. After the backward pass — but before the optimizer updates the master weights — you divide the gradients back down by S (in FP32) to undo the scaling. The math of the update is unchanged; you have merely borrowed some of FP16’s unused high-end range to protect the low end. Modern implementations use dynamic loss scaling: start with a large S, and if a step produces inf/NaN gradients (you scaled too far and overflowed) skip that step and halve S; if many steps pass cleanly, double it.

Loss scaling: a worked example with numbers

Follow one small gradient through the process. Suppose a parameter’s true gradient is g = 2.0e-7. In raw FP16 this is a subnormal so poorly resolved it is essentially noise, and slightly smaller siblings flush to zero outright. Choose a scale S = 1024 (2^10, always a power of two so the rescale is exact):

true gradient           g       = 2.0e-7        (underflows / near-zero in FP16)
scale the loss          S       = 1024 = 2^10
scaled gradient         g * S   = 2.0e-7 * 1024 = 2.05e-4   (a healthy FP16 normal)
  -> stored in FP16 with full 10-bit mantissa precision, no underflow

after backward, before optimizer step (do this in FP32):
unscaled gradient       (g*S)/S = 2.05e-4 / 1024 = 2.0e-7   (recovered)
optimizer update        w_master(FP32) -= lr * 2.0e-7

Scaling lifted 2.0e-7 to 2.05e-4, comfortably above FP16’s 6.1e-5 normal floor, so it is stored with real precision instead of being rounded to zero. The unscale restores the true magnitude in FP32 before it touches the master weights, so convergence is identical to full precision. The only tuning knob is S: too small and small gradients still underflow; too large and big gradients overflow to inf — exactly the tension dynamic loss scaling automates by hunting for the largest S that does not overflow.

Why BF16 usually skips loss scaling

Here is the payoff of BF16’s design. Loss scaling exists solely to work around FP16’s narrow range — the fact that small gradients fall off the bottom of the format. BF16 shares FP32’s 8-bit exponent, so its smallest representable magnitude is around 1e-38, the same as FP32. A gradient of 2e-7, or 2e-20 for that matter, sits comfortably inside BF16’s range and does not underflow. There is simply nothing for loss scaling to rescue.

So BF16 training typically runs with no loss scaling at all: no scale factor to tune, no dynamic-scaling state machine, no skipped steps when the scale overshoots, and symmetric protection against overflow on the high end too (BF16 will not hit inf anywhere FP32 would not). What you pay for this simplicity is BF16’s coarse mantissa — individual values are rounder — but for training that noise is tolerable and largely averages out, whereas a gradient rounded to zero is information destroyed forever. This is the practical reason the field has broadly migrated from FP16-plus-loss-scaling to plain BF16 wherever the hardware supports it: same speed and memory win, dramatically less numerical machinery, and fewer ways for a run to silently diverge.

Which operations must stay in FP32

‘Mixed’ precision means a curated list of operations is kept in FP32 even while the bulk of the matmuls run in 16 bits. The rule of thumb: anything that sums many terms or exponentiates stays in FP32, because those are where small rounding errors compound into large ones. The usual FP32 residents:

Operation	Why it stays FP32
Matmul accumulation	Products are summed over the contraction dim; the accumulator overflows the mantissa if kept 16-bit. Inputs are 16-bit, the running sum is FP32.
Softmax	Uses `exp()` and a normalizing sum; exponentials and their reduction lose precision or overflow in 16-bit.
LayerNorm / RMSNorm	Computes a mean and variance — sums of squares — that need FP32 to be stable.
Loss / cross-entropy	`log`/`exp` over the vocabulary; done in FP32 for a trustworthy scalar loss.
Master weights + optimizer state	Momentum and variance accumulators, and the weight ledger, need FP32 to accumulate tiny updates.

The unifying theme is reductions: whenever you add up a long list of numbers, doing it in 16 bits lets the running total’s ulp grow until later small terms vanish into it. Hardware low-precision matrix units already know this — they multiply 16-bit inputs but accumulate into an FP32 register — and frameworks’ automatic-mixed-precision (‘autocast’) lists encode exactly this division of labor: cheap, well-conditioned elementwise and matmul work in 16-bit, delicate reductions and transcendentals in FP32.