Why architecture matters here
FP8 matters because it's the current frontier of low-precision for both training and inference -- delivering major speed and memory gains via native hardware support, with better range-handling than INT8 -- and it's increasingly standard for large models. The push to fewer bits (speed, memory) reached FP8, and it's significant for several reasons: native tensor-core support (Hopper, Blackwell -- FP8 matrix multiplication far faster than BF16, a major speedup), use for both training and inference (FP8 training for faster/cheaper training, FP8 inference for faster/smaller serving), and better range-handling than INT8 (the floating-point dynamic range handling transformers' wide, outlier-prone value distributions better -- often with less accuracy loss than INT8). So FP8 is the current low-precision frontier (used for large-model training and inference -- major speed/memory gains), and understanding it (the formats, scaling, training/inference use, accuracy management) is understanding an increasingly standard technique for efficient large models. For training or serving large models efficiently, FP8 is increasingly important, and understanding it is understanding the low-precision frontier.
The float-vs-int (dynamic range) distinction is a crucial insight, and it's why FP8 often beats INT8 for transformers. INT8 (8-bit integer) has uniform steps (evenly-spaced values -- the same absolute precision across the range) and a fixed range (determined by the scale). FP8 (8-bit float) has a sign, exponent, and mantissa -- so it has dynamic range (the exponent letting it represent both very small and very large values -- with more precision for small values and coarser precision for large -- like all floating point). This matters for transformers because their value distributions are wide and outlier-prone (some activations/weights have large outliers, most values small) -- and FP8's dynamic range handles this better (representing both the small majority and the large outliers -- with the exponent's range) than INT8's uniform steps (which must either clip the outliers or use a coarse scale that loses precision on the small majority -- the outlier problem that plagues INT8 quantization of transformers). So for transformers' outlier-prone distributions, FP8's dynamic range often gives less accuracy loss than INT8 (handling the outliers with the exponent range, rather than clipping or coarsening). This -- FP8's dynamic range (floating point) handling transformers' wide, outlier-prone distributions better than INT8's uniform steps -- is a crucial advantage of FP8 for transformers, and understanding the float-vs-int distinction (dynamic range vs uniform steps) is understanding why FP8 often beats INT8 for these models.
And the range-precision-split (E4M3 vs E5M2) design is a clever detail, matching the format to the use. The two FP8 formats trade range against precision (with 8 bits, more exponent bits means more range but fewer mantissa bits means less precision, and vice versa). E4M3 (4 exponent, 3 mantissa) has more precision (3 mantissa bits) and less range (4 exponent bits) -- suited to the forward pass (weights and activations -- which need precision, and whose range is more bounded). E5M2 (5 exponent, 2 mantissa) has more range (5 exponent bits) and less precision (2 mantissa bits) -- suited to gradients (in the backward pass -- which have a wide dynamic range, some very small and some large, needing the range more than the precision). So the design uses the right format for each use (E4M3 for the precision-needing forward pass, E5M2 for the range-needing gradients) -- matching the format's range/precision tradeoff to what each part of the computation needs. This range-precision split (E4M3 for precision/forward, E5M2 for range/gradients) is a clever detail of FP8 training (using each format where its tradeoff fits), and understanding it is understanding how FP8 matches the format to the computation's needs.
The architecture: every piece explained
Top row: formats and scaling. FP8 formats: the two standard formats -- E4M3 (4 exponent, 3 mantissa -- more precision, less range) and E5M2 (5 exponent, 2 mantissa -- more range, less precision) -- trading range against precision. Float vs int: FP8 (floating point -- dynamic range via the exponent) vs INT8 (integer -- uniform steps, fixed range) -- FP8's dynamic range handling wide, outlier-prone distributions better. Scaling: scale factors (per-tensor or per-block) mapping the values into the FP8 range -- essential with so few bits (using the limited range well); finer scaling (per-block) handling varying value ranges better. Hardware support: native FP8 support in modern tensor cores (Hopper, Blackwell -- FP8 matrix multiplication far faster than BF16) -- the hardware acceleration making FP8 fast.
Middle row: training, inference, format use. FP8 training: mixed-precision training in FP8 -- computing in FP8 (fast) while keeping higher-precision master weights (a high-precision copy of the weights for stable updates) and careful scaling (loss scaling, per-tensor scaling) for stability -- FP8 for the speed, master weights and scaling for the stability. FP8 inference: FP8 weights and activations for inference (faster, smaller) -- serving the model in FP8. Dynamic range (E5M2): E5M2's more range used for gradients (which have a wide dynamic range -- needing the range). Precision (E4M3): E4M3's more precision used for the forward pass (weights, activations -- needing precision) -- the format matched to the use.
Bottom rows: comparison and accuracy. vs INT8 / BF16: FP8 vs INT8 (FP8's dynamic range handling transformers' outliers better -- often less accuracy loss) and vs BF16 (FP8 faster and smaller -- 8 bits vs 16 -- at some precision cost) -- FP8 winning where its range/speed/memory fit (outlier-prone transformers, speed/memory needs). Accuracy management: keeping the model's quality despite the low precision -- via scaling (per-tensor/block), format choice (E4M3/E5M2), keeping sensitive parts higher-precision, and validation -- the central challenge (maintaining accuracy at 8 bits). The ops strip: scaling strategy (the scaling approach -- per-tensor vs per-block, static vs dynamic -- for using the FP8 range well, crucial for accuracy), calibration (calibrating the scales -- from representative data -- for inference, mapping the values into the FP8 range well), and validation (validating the FP8 model's accuracy -- ensuring the low precision hasn't degraded quality unacceptably -- the essential check).
End-to-end flow
Trace FP8 mixed-precision training. A large model is trained with FP8 for speed. The forward and backward passes compute in FP8 (the matrix multiplications in FP8 on the tensor cores -- far faster than BF16) -- using E4M3 for the forward pass (weights and activations -- needing precision -- E4M3's 3 mantissa bits) and E5M2 for the gradients (in the backward pass -- needing the wide dynamic range -- E5M2's 5 exponent bits). For stability, the training keeps higher-precision master weights (a BF16/FP32 copy of the weights -- the FP8 weights used for the fast computation, but the master weights accumulating the updates precisely -- so the weight updates are stable, not degraded by the FP8 precision) and uses careful scaling (loss scaling and per-tensor scaling -- mapping the values into the FP8 range well, avoiding overflow/underflow). So the training is fast (computing in FP8 on the tensor cores) and stable (the master weights and scaling maintaining stability) -- FP8 mixed-precision training delivering the speed (FP8 compute) while managing the stability (master weights, scaling). The format use (E4M3 forward, E5M2 gradients) matched each part's needs (precision vs range).
The INT8-comparison and inference vignettes show FP8's advantages. An INT8-comparison case: the team compares FP8 and INT8 for quantizing a transformer. INT8 struggles with the transformer's outliers (some large-outlier activations -- INT8's uniform steps must either clip them or use a coarse scale losing precision on the small majority -- accuracy loss). FP8 handles the outliers better (its dynamic range -- the exponent -- representing both the small majority and the large outliers -- less accuracy loss) -- so FP8 gives better accuracy than INT8 for the outlier-prone transformer -- the dynamic-range advantage. An inference case: the team serves the model in FP8 (FP8 weights and activations) -- faster (FP8 tensor cores) and smaller (8 bits vs BF16's 16) than BF16 serving -- with the accuracy maintained (via scaling and validation) -- the FP8 inference giving the speed/memory benefit while keeping the quality.
The accuracy-management and validation vignettes complete it. An accuracy-management case: the team manages the FP8 accuracy carefully -- using per-block scaling (finer than per-tensor -- handling the varying value ranges across the tensor better -- more accurate), keeping the most sensitive parts higher-precision (some layers/operations kept in BF16 -- where FP8 would degrade too much), and calibrating the scales from representative data -- maintaining the accuracy despite the 8 bits. A validation case: the team validates the FP8 model's accuracy (measuring its quality on evaluation tasks -- confirming the FP8 hasn't degraded it unacceptably) -- the essential check (ensuring the low precision is acceptable before deploying). The consolidated discipline the team documents: use FP8 for training (mixed precision -- FP8 compute for speed, master weights and scaling for stability) and inference (FP8 weights/activations -- faster, smaller), leverage the native tensor-core support (Hopper/Blackwell -- the major speedup), use the right format for each use (E4M3 for the precision-needing forward, E5M2 for the range-needing gradients), exploit FP8's dynamic-range advantage over INT8 (handling transformers' outliers better), manage the accuracy carefully (scaling -- per-tensor/block, format choice, keeping sensitive parts higher-precision, calibration), and validate the accuracy (ensuring the low precision is acceptable) -- because FP8 is the current low-precision frontier for both training and inference (major speed/memory gains via native hardware support), with better range-handling than INT8 (its floating-point dynamic range suiting transformers' outlier-prone distributions), requiring careful accuracy management (scaling, format choice, validation) to maintain quality at 8 bits.