Why it matters

LLM.int8() is foundational int8 paper. Understanding shapes theory.

Advertisement

The architecture

Row-wise scaling.

Column outlier: keep fp16.

Combine outputs.

LLM.int8() flowRow-wise scaleint8 mainOutlier columnsfp16 separateCombine outputsadd matmul resultsEmergent features (magnitude spikes) drove design; keep outliers full precision
LLM.int8().
Advertisement

How it works end to end

Emergent outlier features at scale.

Row scale for main matrix.

Column subset fp16.

Sum results.