Why it matters

FP8 is next precision frontier. Understanding shapes modern training.

Advertisement

The architecture

E4M3: 4-bit exponent, 3-bit mantissa.

E5M2: 5-bit exponent, 2-bit mantissa.

Per-tensor scaling.

FP8 trainingE4M3weights + actE5M2gradientsPer-tensor scalemanage rangeTransformer Engine on Hopper automates FP8; delayed scaling for stability
FP8.
Advertisement

How it works end to end

Transformer Engine (TE) library.

Delayed scaling: use previous batch stats.

Amax history tracking.

Master weights in fp32.