Why architecture matters here

Activation quantization fails on outliers and missing kernels. Architecture matters because scale + kernel choose together.

Advertisement

The architecture: every piece explained

The top strip is design. Activation tensors. Scale strategy. Outlier isolation. Static vs dynamic.

The middle row is integration. Quantize + dequant. Kernel support. Numerics. Combined with weight quant.

The lower rows are practice. Eval. Metrics. Ops — hardware fit + checkpoint + fallback.

Activation quantization — per-token + per-tensor + outlier handling + kernel supportquantize the values that flow throughActivation tensorsper layerScale strategyper-tensor / per-token / per-channelOutlier isolationkeep fp16 sideStatic vs dynamiccalibrated vs runtimeQuantize + dequantboundaryKernel supportint8/fp8 GEMMNumericsfp32 accumulateCombined with weight quantW4A8 / W8A8Evalquality regressionMetricsthroughput + PPLOps — hardware fit + checkpoint format + fallbackwraprunsafecomposetestmeasuremeasureoperateoperate
Activation quantization strategies and integration.
Advertisement

End-to-end flow

End-to-end: model uses W8A8 with per-token activation scales + outlier isolation for known bad channels. Fused int8 GEMM kernel executes. Eval passes. Throughput 2x fp16.