Why it matters
bnb int8 makes big models loadable. Understanding shapes quant choice.
Advertisement
The architecture
Row-wise scaling per weight.
Outlier features kept fp16.
Two matmuls combined.
Advertisement
How it works end to end
load_in_8bit=True in HF.
Quality: negligible drop.
Memory: ~50%.
Speed: often slower than fp16.