Why it matters

bitsandbytes made INT8 loading trivial in the transformers ecosystem. Understanding LLM.int8() explains why 'load_in_8bit=True' works well.

Advertisement

The architecture

Identify outlier columns: features with values above threshold (~6).

Split matmul: outliers in FP16, non-outliers in INT8. Sum results.

LLM.int8() flowIdentify outliers|x| > thresholdSplit matmulFP16 + INT8Sum resultscombined outputOnly ~0.1% of features are outliers; separating them protects quality
LLM.int8() decomposition.
Advertisement

How it works end to end

Threshold: ~6 by default. Higher threshold = fewer outliers, faster, but more quality loss.

Overhead: FP16 computation for outliers has some cost but is small (few features).

Model support: bitsandbytes integrates with transformers via load_in_8bit=True.