BF16 (Brain Float), in depth: the format as a storage, inference and quantization baseline

The format from first principles

A BF16 value is 16 bits: 1 sign bit, 8 exponent bits and 7 stored mantissa bits. Like FP32 it represents (−1)^s × 2^e−127 × 1.m for normal numbers, with the same exponent bias. In fact BF16 is exactly the upper 16 bits of an FP32 word, which is the whole design: same range as FP32, much less precision, and conversion that is just a rounding of the low half.

The table explains BF16's role. Its range means values that fit in FP32 almost never overflow or underflow when converted, unlike FP16, which overflows above 65504. Its precision is about two to three significant decimal digits, which neural network weights and activations tolerate well in matrix inputs but which is visibly coarse for sums, counters and positions. Upcasting BF16 to FP32 is exact (append 16 zero bits), so every BF16 value has a perfect FP32 twin.

Converting FP32 to BF16 correctly

There are two ways to drop the low 16 bits. Truncation just discards them; it is biased toward zero and its error reaches a full unit in the last place. Round to nearest, ties to even (RNE) rounds to the closer BF16 value and has a maximum relative error of 2^-8, about 0.39%. Frameworks use RNE for tensor conversion; hand-written kernels and file converters sometimes truncate, which is a quiet source of bias. NaN needs special handling, because rounding a NaN's mantissa can carry into the exponent and produce infinity.

import numpy as np

def fp32_to_bf16_bits(x):
    """Round-to-nearest-even FP32 -> BF16; returns uint16 bit patterns."""
    u = np.asarray(x, dtype=np.float32).view(np.uint32)
    lsb = (u >> 16) & 1                       # ties go to the even result
    rounded = (u + 0x7FFF + lsb) >> 16
    nan = np.isnan(np.asarray(x, dtype=np.float32))
    out = np.where(nan, (u >> 16) | 0x0040, rounded)   # keep NaN a quiet NaN
    return out.astype(np.uint16)

def bf16_bits_to_fp32(b):
    return (np.asarray(b, dtype=np.uint32) << 16).view(np.float32)   # exact

x = np.array([1.0, 3.14159265, 0.1], dtype=np.float32)
print(bf16_bits_to_fp32(fp32_to_bf16_bits(x)))    # 1.0, 3.140625, 0.10009766

Two worked encodings make the rounding concrete. The FP32 pattern of π is 0x40490FDB. Its low half, 0x0FDB, is below 0x8000, so RNE keeps the upper half: 0x4049, which decodes to sign 0, exponent 128 (a factor of 2), mantissa 0x49 = 73/128, giving 1.5703125 × 2 = 3.140625. The FP32 pattern of 0.1 is 0x3DCCCCCD; its low half 0xCCCD is above 0x8000, so RNE rounds up to 0x3DCD = 1.6015625 × 2^-4 = 0.10009765625. Truncation would give 0x3DCC = 0.099609375: an error four times larger in this case, and always in the same direction.

In PyTorch, tensor.to(torch.bfloat16) rounds to nearest even. If you write a custom converter, test it against the framework on random inputs, on exact ties, and on NaN and infinity before you trust a converted checkpoint.

When the source checkpoint is FP32, measure what the conversion cost per tensor instead of assuming it is negligible. The relative error should sit near the RNE average, roughly a tenth to a few tenths of a percent; a tensor far above that usually holds tiny values next to large ones, and is a candidate to keep in FP32:

import torch

def bf16_conversion_report(state_dict, worst=10):
    rows = []
    for name, t in state_dict.items():
        if t.dtype != torch.float32:
            continue
        back = t.to(torch.bfloat16).float()
        rel = ((back - t).norm() / t.norm().clamp_min(1e-30)).item()
        rows.append((rel, name, tuple(t.shape)))
    return sorted(rows, reverse=True)[:worst]

FP16 and BF16 are not interchangeable

Moving between the two 16-bit formats loses something in both directions. FP16 to BF16 discards 3 mantissa bits, so an FP16-trained model can lose accuracy it relied on. BF16 to FP16 keeps more mantissa but has a much smaller range: any value above 65504 becomes infinity, and values below FP16's subnormal floor (about 6e-8) become zero. Activations in large language models often contain outlier channels with large magnitudes, which is exactly where FP16 overflow shows up, so a model that serves fine in BF16 may produce NaNs in FP16. Check before converting:

import torch

def fp16_hazards(state_dict, top=10):
    report = []
    for name, t in state_dict.items():
        if not t.is_floating_point():
            continue
        a = t.float().abs()
        over = (a > 65504).sum().item()
        under = ((a > 0) & (a < 6e-8)).sum().item()
        if over or under:
            report.append((name, over, under, a.max().item()))
    return sorted(report, key=lambda r: -r[1])[:top]

Weights rarely trip this check; activations are the real risk, so run a few representative prompts with forward hooks recording per-layer maximum absolute activations before switching a BF16 model to FP16 serving.

Where inference must not run in BF16

BF16 is safe as the input to matrix multiplications because tensor cores accumulate the products in FP32 (see tensor core architecture). The danger is any operation that sums many terms, compares close values, or represents an exact count. Keep these in FP32:

• Softmax. Subtracting the row maximum and summing exponentials over thousands of keys needs FP32; fused attention kernels do this internally.
• Normalisation statistics. The mean and variance in LayerNorm and RMSNorm are sums over the hidden dimension.
• Final logits and sampling. Top-p and temperature compare small probability differences; a BF16 logit has a spacing of 0.0625 near 10 and 0.125 near 20, coarse enough to change which tokens survive the cut.
• Position and angle arithmetic. BF16 represents integers exactly only up to 256. Between 512 and 1024 its spacing is 4, so positions 1003 and 1005 both round to 1004. If rotary embedding angles are computed from BF16 positions, nearby tokens get identical angles. Compute positions and angles in FP32, then cast the cos and sin tables.
• Long accumulations in your own code. A running sum in BF16 stops growing once it is roughly 256 times larger than the increment.

A consequence worth knowing before you file a bug: greedy decoding in BF16 and in FP32 will diverge after some number of tokens, because a tiny difference eventually flips an argmax. That is expected. Compare logits with dtype-aware tolerances instead; torch.testing.assert_close already uses looser default tolerances for bfloat16 than for float32.

BF16 as the quantization baseline

When a model is released in BF16, the BF16 weights are the ground truth. Quantization methods covered elsewhere on this site, from FP8 to the per-layer plans in mixed-precision inference, should start from those weights and be judged against BF16 outputs, for three reasons.

1. Upcasting BF16 to FP32 for calibration is lossless, so computing scales and Hessians in FP32 introduces no extra conversion error.
2. Converting a BF16 checkpoint to FP32 and back is a no-op; converting BF16 to FP16 first and then quantizing stacks two lossy steps and can overflow.
3. The alternative your users would otherwise receive is the BF16 model, so that is the comparison that answers whether quantization cost them anything.

def compare_to_reference(model_bf16, model_q, batches):
    """Per-batch KL divergence of quantized vs BF16 next-token distributions."""
    kls = []
    with torch.no_grad():
        for ids in batches:
            p_ref = torch.log_softmax(model_bf16(ids).logits.float(), -1)
            p_q = torch.log_softmax(model_q(ids).logits.float(), -1)
            kls.append(torch.nn.functional.kl_div(p_q, p_ref, log_target=True,
                                                  reduction="batchmean").item())
    return sum(kls) / len(kls)

Note the .float() before the softmax in both branches: the comparison itself must not run in BF16.

Hardware and software support

BF16 originated at Google for TPUs. NVIDIA tensor cores support it from the Ampere generation (A100) onward; on older GPUs such as V100 and T4 there is no BF16 tensor-core path, so use FP16 there. On CPUs, Intel added AVX-512 BF16 instructions with Cooper Lake and AMX tiles with Sapphire Rapids, and Arm added BF16 instructions in Armv8.6-A. Support varies by exact SKU, so detect it at runtime rather than assuming: in PyTorch, torch.cuda.is_bf16_supported() answers for the current GPU.

Elementwise kernels for BF16 commonly load BF16, compute in FP32 registers and store BF16, so the speed benefit is mostly memory bandwidth, and the precision loss happens at each store. A chain of unfused elementwise ops rounds at every step; fusion therefore improves both speed and accuracy.

Worked example: sizing a BF16 deployment

Take an 8 billion parameter decoder with 32 layers, 8 key-value heads and a head dimension of 128. Weights: 8e9 × 2 bytes = 16 GB in BF16, against 32 GB in FP32 and roughly 4 to 5 GB with 4-bit weights plus scales. KV cache per token: 2 (keys and values) × 32 layers × 8 heads × 128 × 2 bytes = 131,072 bytes, or 128 KiB. An 8,192 token context therefore needs 1 GiB per sequence, and 16 concurrent sequences need 16 GiB, as much as the weights.

That arithmetic is why BF16 is usually the starting point rather than the end state: on a 24 GB card the model fits but leaves room for very little cache, which pushes you toward quantized weights, a quantized KV cache, or both, each measured against the BF16 reference.

Failure modes

Trade-offs

What to do next

1. Find out which dtype your model was trained and released in, and keep that file as the reference.
2. Convert with the framework's RNE path, and test any custom converter on ties, NaN and infinity.
3. Audit your inference graph for softmax, norms, logits, positions and accumulations, and keep them in FP32.
4. Before any FP16 deployment, record per-layer activation maxima on real prompts.
5. Size weights and KV cache with the arithmetic above to decide whether you need quantization at all.
6. Measure every quantized variant against BF16 outputs with KL divergence and task metrics, not against FP32.