Mixed-precision training is the art of doing most of the arithmetic in a 16-bit float to go faster and use half the memory, while keeping just enough 32-bit state around that the model still converges as if you had trained in full precision the whole time. The whole subject turns on one fact about floating-point numbers: the bits you spend on the exponent buy you range, and the bits you spend on the mantissa buy you precision, and a 16-bit float simply does not have enough of either to spare. FP16 and BF16 make opposite cuts — FP16 keeps precision and loses range, BF16 keeps range and loses precision — and that single design choice decides whether you need loss scaling, how your gradients behave, and which format your CPU actually wants to run. This piece builds the whole picture from the bit layout up: the three formats, the FP32 master copy, loss scaling with real numbers, the ops that must stay in FP32, and why BF16 is the format a small language model on a CPU should reach for.
Anatomy of a floating-point number
Every IEEE-style float is three fields packed into a fixed number of bits: a sign bit, an exponent field, and a mantissa (also called the significand or fraction). The value is reconstructed as (-1)^sign × 1.mantissa × 2^(exp - bias). The mantissa is the significant digits; the exponent slides the binary point to position those digits anywhere from the very small to the very large.
This split is the key to everything that follows. The exponent width sets the range — how big and how small a number you can represent before you overflow to infinity or underflow to zero. The mantissa width sets the precision — how finely you can distinguish two nearby numbers, i.e. the size of the gap (the ulp, unit in the last place) between one representable value and the next. Critically, precision is relative: near 1.0 a format with an m-bit mantissa has a gap of 2^-m, and near 1000 the gap is 2^-m × 1024. A float is a ruler whose tick spacing grows with the magnitude of what you are measuring. When you shrink a 32-bit float to 16 bits, you must decide which of those two budgets — range or precision — to cut. That decision is the entire story of FP16 versus BF16.
FP32: the full-precision baseline
FP32 (IEEE single precision) is the reference everyone else is measured against: 1 sign bit, 8 exponent bits, 23 mantissa bits. The 8-bit exponent gives a dynamic range of roughly ±1.2e-38 up to ±3.4e38, and the 23-bit mantissa (24 bits of effective significand with the implicit leading 1) gives about 7 decimal digits of precision — a relative gap near 1.0 of 2^-23 ≈ 1.2e-7.
FP32 [ S | EEEEEEEE | MMMMMMMMMMMMMMMMMMMMMMM ] 32 bits
1 8 23
range ~ +/- 3.4e38 precision ~ 7 decimal digitsFor decades this was simply ‘what training runs in.’ It has enough range that activations, weights, and gradients essentially never overflow, and enough precision that the tiny weight updates of gradient descent actually stick. The problem is cost: every FP32 tensor is 4 bytes per element, and every FP32 matmul moves and multiplies those 4-byte values. Since modern training and inference are dominated by moving numbers through memory and multiply-accumulate units, FP32 is leaving a large factor of speed and memory on the table — if only you could do the heavy arithmetic in half the bits without breaking convergence. That ‘if only’ is what mixed precision delivers.
FP16: keep the precision, lose the range
FP16 (IEEE half precision) is the intuitive way to halve a float: 1 sign bit, 5 exponent bits, 10 mantissa bits. It spends its scarce bits on the mantissa, so near 1.0 it still resolves about 3–4 decimal digits (a gap of 2^-10 ≈ 9.8e-4). What it sacrifices is range: the 5-bit exponent caps the largest normal value at 65504 and puts the smallest normal number at about 6.1e-5 (2^-14), with subnormals limping down to about 6e-8 (2^-24) before flushing to zero.
FP16 [ S | EEEEE | MMMMMMMMMM ] 16 bits
1 5 10
max normal 65504 smallest normal ~ 6.1e-5
subnormals reach ~ 6e-8, then underflow to 0That narrow range is the source of nearly every FP16 headache. Gradients in a deep network are routinely far below 6e-5 — especially early in training and in the lower layers — so a large fraction of them round to exactly zero, and zero gradients mean those parameters simply stop learning. On the high end, a few large activations or an unlucky attention score can punch through 65504 and produce an inf that poisons the whole backward pass. FP16 works, but only with a crutch — loss scaling — to drag the gradients back into representable range.
BF16: keep the range, lose the precision
BF16 (bfloat16, Google Brain’s format) makes the opposite cut: 1 sign bit, 8 exponent bits, 7 mantissa bits. The exponent is identical to FP32’s, so BF16 has the same enormous dynamic range — roughly ±3.4e38 — and essentially never overflows or underflows on values that FP32 would have handled. The price is precision: with only 7 mantissa bits, the gap near 1.0 is 2^-7 ≈ 7.8e-3, giving barely 2–3 decimal digits. BF16 is a coarse ruler with a very long reach.
BF16 [ S | EEEEEEEE | MMMMMMM ] 16 bits
1 8 7
range ~ +/- 3.4e38 (same as FP32) precision ~ 2-3 digits
FP32 [ S | EEEEEEEE | MMMMMMMMMMMMMMMMMMMMMMM ]
BF16 [ S | EEEEEEEE | MMMMMMM ] <-- just the top 16 bits of FP32Notice the beautiful consequence of matching FP32’s exponent: a BF16 value is very nearly just the top 16 bits of the corresponding FP32 value. Converting FP32→BF16 is close to truncation, and BF16→FP32 is close to zero-padding. That near-trivial conversion, plus the fact that the range problems that force loss scaling in FP16 simply do not occur, is why BF16 has become the default low-precision training format — and, as we will see, why it is especially at home on a CPU.
Range versus precision: the trade in one table
Put the three formats side by side and the exponent/mantissa trade is stark. The exponent column governs whether your numbers survive at all; the mantissa column governs how accurately they are recorded.
| Format | Sign | Exponent | Mantissa | Max value | Rel. precision |
|---|---|---|---|---|---|
| FP32 | 1 | 8 | 23 | ~3.4e38 | ~1.2e-7 (~7 digits) |
| FP16 | 1 | 5 | 10 | 65504 | ~9.8e-4 (~3-4 digits) |
| BF16 | 1 | 8 | 7 | ~3.4e38 | ~7.8e-3 (~2-3 digits) |
Read across the rows and the philosophy of each format is obvious. FP16 and BF16 both weigh 16 bits, but FP16 buys three extra mantissa bits by giving up three exponent bits, and BF16 does the reverse. The deep-learning verdict, learned the hard way, is that range matters more than precision for training neural networks. Networks are remarkably tolerant of noisy, low-precision weights and activations — stochastic gradient descent is itself a noisy process, and rounding error just adds a little more — but they are not tolerant of gradients silently becoming zero or activations becoming infinity. BF16 protects the thing that matters (range) and spends the thing the network can absorb (precision). That is why, on hardware that supports it, BF16 generally trains more robustly and with less babysitting than FP16.
Why compute in low precision at all?
The motivation is entirely about memory and speed, and both come from the same fact: a 16-bit value is half the bytes of a 32-bit value. On the memory side, activations stored for the backward pass, the largest transient cost of training, halve. A transformer’s activation memory scales with batch × sequence × hidden × layers; cutting the per-element size from 4 bytes to 2 lets you fit roughly twice the batch or twice the context in the same footprint. Weights and gradients moving across the memory bus also halve.
On the speed side, most training and inference is bandwidth-bound: the arithmetic units sit idle waiting for numbers to arrive from memory. Halving the bytes per number roughly doubles the effective bandwidth and doubles how much of a weight matrix fits in cache. On top of that, dedicated low-precision matrix units — GPU tensor cores, and on CPUs the AVX-512 BF16 and AMX instructions — execute 16-bit multiply-accumulate at two to several times the throughput of FP32. The combined effect is commonly a 1.5–3× training speedup and a ~2× memory reduction for the low-precision tensors. The catch is that you cannot naively cast everything to 16 bits and hope for the best — some quantities are too delicate. Managing which stays high-precision is the ‘mixed’ in mixed precision.
The FP32 master copy of the weights
The first quantity too delicate to demote is the weights themselves — specifically, the copy that gets updated. Mixed precision keeps a full-precision FP32 master copy of every weight, and the optimizer applies its updates to that master copy. Each step, the master weights are cast down to BF16 (or FP16) for the forward and backward passes, but the authoritative state that persists across steps lives in FP32.
The reason is update swamping. Gradient-descent updates are typically minuscule relative to the weights they modify — a weight near 1.0 might receive an update of 1e-4 or smaller. If the weight is stored in 16 bits, that update can be smaller than a single ulp at the weight’s magnitude, so weight + update rounds right back to weight and the step is silently lost. Over thousands of steps those lost updates are the difference between learning and stalling. Keeping the master copy in FP32 — with its ~1.2e-7 resolution near 1.0 — means even tiny updates accumulate correctly. Note this bites BF16 harder than FP16, because BF16’s coarser 7-bit mantissa has an even larger ulp; the master copy is not an FP16-only patch, it is fundamental to both.
Update swamping: a worked example
Make the swamping concrete. Take a weight w = 1.0 and a per-step update Δw = -0.0001 (learning rate times gradient). The spacing of representable values near 1.0 — the ulp — is 2^-m for an m-bit mantissa:
ulp at 1.0:
FP32 2^-23 ~ 1.2e-7 BF16 2^-7 ~ 7.8e-3 FP16 2^-10 ~ 9.8e-4
update |dw| = 1e-4
BF16: 1e-4 < half-ulp (3.9e-3) -> 1.0 - 1e-4 rounds to 1.0 (LOST)
FP16: 1e-4 < half-ulp (4.9e-4) -> 1.0 - 1e-4 rounds to 1.0 (LOST)
FP32: 1e-4 >> half-ulp (6e-8) -> 1.0 - 1e-4 = 0.9999 (KEPT)In both 16-bit formats the update is smaller than half the gap to the next representable number, so rounding-to-nearest snaps the result straight back to 1.0: the optimizer ‘stepped’ but nothing moved. Only the FP32 master copy records the change. Now run 10,000 steps of -1e-4 each: in FP32 the weight drifts to 0.0 as it should; in a 16-bit-only weight it might never leave 1.0, because each individual step keeps vanishing before it can accumulate. This is precisely why the master copy exists — it is the ledger with enough decimal places to record small deposits that the 16-bit working copy would round away.
Loss scaling: rescuing FP16 gradients from underflow
The FP32 master copy fixes the update side. FP16 has a second, separate problem on the gradient side: many gradient values are so small they fall below FP16’s minimum representable magnitude and underflow to zero before the optimizer ever sees them. Recall FP16’s smallest normal is ~6.1e-5 and it flushes to zero below ~6e-8. Activation gradients in real networks routinely live in the 1e-7 to 1e-10 region — a whole swath of them simply disappears.
The fix is loss scaling, and it is delightfully simple. Multiply the loss by a large constant S before calling backward. By the chain rule, every gradient in the network is then multiplied by the same S, shifting the entire gradient distribution up into FP16’s representable range where it survives. After the backward pass — but before the optimizer updates the master weights — you divide the gradients back down by S (in FP32) to undo the scaling. The math of the update is unchanged; you have merely borrowed some of FP16’s unused high-end range to protect the low end. Modern implementations use dynamic loss scaling: start with a large S, and if a step produces inf/NaN gradients (you scaled too far and overflowed) skip that step and halve S; if many steps pass cleanly, double it.
Loss scaling: a worked example with numbers
Follow one small gradient through the process. Suppose a parameter’s true gradient is g = 2.0e-7. In raw FP16 this is a subnormal so poorly resolved it is essentially noise, and slightly smaller siblings flush to zero outright. Choose a scale S = 1024 (2^10, always a power of two so the rescale is exact):
true gradient g = 2.0e-7 (underflows / near-zero in FP16)
scale the loss S = 1024 = 2^10
scaled gradient g * S = 2.0e-7 * 1024 = 2.05e-4 (a healthy FP16 normal)
-> stored in FP16 with full 10-bit mantissa precision, no underflow
after backward, before optimizer step (do this in FP32):
unscaled gradient (g*S)/S = 2.05e-4 / 1024 = 2.0e-7 (recovered)
optimizer update w_master(FP32) -= lr * 2.0e-7Scaling lifted 2.0e-7 to 2.05e-4, comfortably above FP16’s 6.1e-5 normal floor, so it is stored with real precision instead of being rounded to zero. The unscale restores the true magnitude in FP32 before it touches the master weights, so convergence is identical to full precision. The only tuning knob is S: too small and small gradients still underflow; too large and big gradients overflow to inf — exactly the tension dynamic loss scaling automates by hunting for the largest S that does not overflow.
Why BF16 usually skips loss scaling
Here is the payoff of BF16’s design. Loss scaling exists solely to work around FP16’s narrow range — the fact that small gradients fall off the bottom of the format. BF16 shares FP32’s 8-bit exponent, so its smallest representable magnitude is around 1e-38, the same as FP32. A gradient of 2e-7, or 2e-20 for that matter, sits comfortably inside BF16’s range and does not underflow. There is simply nothing for loss scaling to rescue.
So BF16 training typically runs with no loss scaling at all: no scale factor to tune, no dynamic-scaling state machine, no skipped steps when the scale overshoots, and symmetric protection against overflow on the high end too (BF16 will not hit inf anywhere FP32 would not). What you pay for this simplicity is BF16’s coarse mantissa — individual values are rounder — but for training that noise is tolerable and largely averages out, whereas a gradient rounded to zero is information destroyed forever. This is the practical reason the field has broadly migrated from FP16-plus-loss-scaling to plain BF16 wherever the hardware supports it: same speed and memory win, dramatically less numerical machinery, and fewer ways for a run to silently diverge.
Which operations must stay in FP32
‘Mixed’ precision means a curated list of operations is kept in FP32 even while the bulk of the matmuls run in 16 bits. The rule of thumb: anything that sums many terms or exponentiates stays in FP32, because those are where small rounding errors compound into large ones. The usual FP32 residents:
| Operation | Why it stays FP32 |
|---|---|
| Matmul accumulation | Products are summed over the contraction dim; the accumulator overflows the mantissa if kept 16-bit. Inputs are 16-bit, the running sum is FP32. |
| Softmax | Uses exp() and a normalizing sum; exponentials and their reduction lose precision or overflow in 16-bit. |
| LayerNorm / RMSNorm | Computes a mean and variance — sums of squares — that need FP32 to be stable. |
| Loss / cross-entropy | log/exp over the vocabulary; done in FP32 for a trustworthy scalar loss. |
| Master weights + optimizer state | Momentum and variance accumulators, and the weight ledger, need FP32 to accumulate tiny updates. |
The unifying theme is reductions: whenever you add up a long list of numbers, doing it in 16 bits lets the running total’s ulp grow until later small terms vanish into it. Hardware low-precision matrix units already know this — they multiply 16-bit inputs but accumulate into an FP32 register — and frameworks’ automatic-mixed-precision (‘autocast’) lists encode exactly this division of labor: cheap, well-conditioned elementwise and matmul work in 16-bit, delicate reductions and transcendentals in FP32.
The full mixed-precision training loop
Assemble the pieces into the loop that runs every step. The master weights live in FP32; a 16-bit working copy is what the compute-heavy passes actually use.
for each step:
1. cast FP32 master weights -> BF16/FP16 working weights
2. FORWARD in 16-bit (matmuls fast; softmax/norm/loss internally FP32)
3. compute LOSS in FP32
(FP16 only) loss *= S # loss scaling to lift small gradients
4. BACKWARD in 16-bit (accumulate into FP32; grads are 16-bit tensors)
(FP16 only) grads /= S # unscale, in FP32
5. optimizer step: update FP32 master weights (+ FP32 momentum/variance)
BF16 path: no scale factor anywhereTwo design decisions carry the whole scheme. First, the expensive, well-conditioned work (the big matmuls in forward and backward) runs in 16 bits — that is where the speed and activation-memory savings come from. Second, the delicate, cheap, cumulative state (master weights, optimizer moments, reductions) stays in FP32 — that is what preserves convergence. The BF16 path is strictly simpler than the FP16 path: delete steps 3b and 4b, the loss-scaling lines. That deletion — no scale to tune, no overflow-detection state machine — is a real reduction in operational surface area, and it is a large part of why practitioners reach for BF16 the moment their hardware offers it.
BF16 on the CPU: the natural fit for small language models
Mixed precision was born on GPUs, but its logic transfers cleanly to CPUs, and there BF16 has a special advantage. Modern server and desktop CPUs now include native BF16 acceleration — Intel’s AVX-512 BF16 and the AMX tile engine, and ARM’s BF16 extensions — that multiply BF16 operands while accumulating into FP32, exactly the mixed-precision pattern. FP16, by contrast, historically had far weaker CPU support; on many CPUs it is a storage format that must be converted to compute, not a first-class arithmetic type.
BF16’s FP32-shaped exponent pays off again here: converting FP32↔BF16 is essentially truncating or zero-padding the low 16 bits, so a CPU kernel can move between the master copy and the working copy almost for free, with no expensive range remapping and no loss-scaling bookkeeping. For a small language model (SLM) running on a CPU — the recurring theme of this series — that matters twice over. CPU inference and fine-tuning are overwhelmingly memory-bandwidth-bound, so storing weights and activations in BF16 halves the bytes crossing the bus and doubles the fraction of the model that fits in cache, often the single biggest lever on tokens-per-second. And BF16’s no-loss-scaling robustness means you can fine-tune a small model on a laptop CPU without the numerical babysitting FP16 would demand. BF16 is, in short, the low-precision format the CPU actually wants.
Common pitfalls and how to avoid them
Mixed precision fails in a handful of recognizable ways. Casting the master weights to 16-bit permanently — forgetting to keep the FP32 copy — reintroduces update swamping and the model quietly stops learning; symptoms are a loss that plateaus early for no obvious reason. Doing a reduction in low precision — a hand-written softmax, norm, or sum that accumulates in 16-bit — produces subtly wrong results or creeping divergence; keep accumulators FP32. Choosing a bad loss scale in FP16 gives you either persistent underflow (scale too low, gradients still zero) or a flood of inf/NaN (scale too high); prefer dynamic loss scaling over a hand-tuned constant.
Two more worth naming. Assuming BF16 precision is free: its 2–3 digits are fine for training but can bite at inference for accumulation-heavy or ill-conditioned computations — when in doubt, evaluate in FP32 or keep the sensitive layer high-precision. And expecting FP16 and BF16 to be interchangeable: they are not — FP16 needs loss scaling and BF16 does not, FP16 has finer resolution and BF16 has vastly more range, and code that silently swaps one for the other will behave differently. The safe default on modern hardware is straightforward: BF16 compute, FP32 master weights and reductions, and no loss scaling — the simplest configuration that captures nearly all of the speed and memory win.