Why architecture matters here
Calibration matters because it largely determines whether quantization preserves accuracy or destroys it -- choosing the quantization ranges well (from representative data) is the key to accurate post-training quantization. Quantization maps floats to a small integer range, and the range choice is crucial: a poor range (too wide -- wasting precision on common values; too narrow -- clipping extremes) loses information (degrading accuracy), while a good range (well-chosen -- balancing coverage and precision) preserves accuracy. Calibration is how the ranges are chosen (from representative data -- observing the actual value distributions) -- so good calibration is the difference between quantization preserving accuracy (usable) and destroying it (unusable). This is especially important for post-training quantization (PTQ -- quantizing a trained model without retraining, using calibration data) -- the common, convenient approach, where calibration is the key step. For quantizing models (increasingly important for efficient deployment), calibration is central to preserving accuracy, and understanding it (how to choose ranges from representative data) is understanding how to quantize accurately.
The representative-data insight is foundational, and it's why calibration data matters so much. Calibration chooses the quantization ranges based on the actual distribution of values -- and for activations (which depend on the input), the distribution is observed by running calibration data through the model. So the calibration data must be representative of the real inputs (the data the model will see in production) -- because the ranges are chosen to fit the calibration data's distributions, and if the calibration data isn't representative (e.g., too small, or a different distribution than production), the ranges will be wrong for the real inputs (chosen for the unrepresentative calibration data -- so production inputs, with different distributions, are poorly quantized -- outliers not covered, or ranges mis-fit). So representative calibration data is essential (the ranges are only as good as the data they're calibrated on). This is a common pitfall (using unrepresentative or too-little calibration data -- leading to poor ranges and accuracy loss on real inputs). The calibration data should be a representative sample of the real inputs (enough data, from the real distribution -- so the observed distributions, and thus the chosen ranges, match production). Understanding the representative-data requirement (the calibration data must represent the real inputs -- since the ranges are calibrated on it) is understanding a foundational aspect of calibration.
And the outlier-handling reality is the crux of range selection, because outliers force a precision tradeoff. Neural network value distributions often have outliers -- most values in a small range, with a few extreme values far outside it. This creates a dilemma for range selection. If you use the full min/max range (covering the outliers), the range is very wide (to include the extreme outliers) -- so the common values (the small range where most values are) get squeezed into few integer levels (wasting precision -- most of the integer levels are 'used up' on the sparse outlier region, leaving few for the dense common region) -- poor precision for the common values (accuracy loss). If you use a narrower range (e.g., percentile -- clipping the extreme tails, covering, say, the 99.9th percentile), the common values get more precision (the range fits them better -- more integer levels for the dense region), but the outliers are clipped (values beyond the range saturate to the range's edge -- losing the outlier information). So there's a tradeoff: cover the outliers (min/max -- but waste precision on common values) vs clip them (percentile -- better precision for common values, but lose the outliers). The right choice depends on the importance of the outliers (if they matter, cover them; if they're noise, clip them for better common-value precision) -- and often, clipping (percentile) gives better overall accuracy (the common values, being the majority, benefit more from the precision than the rare outliers lose from clipping). This outlier handling (the min/max-vs-percentile tradeoff -- covering vs clipping outliers, balancing outlier preservation against common-value precision) is the crux of range selection, and it's why calibration methods matter. Understanding the outlier-handling tradeoff (cover vs clip, balancing precision) is understanding the crux of quantization range selection.
The architecture: every piece explained
Top row: the problem and data. The problem: mapping floats to a small integer range -- the range choice determining the precision/clipping tradeoff. Calibration data: a representative sample of inputs (run through the model to observe the value distributions -- especially activations, which are input-dependent) -- the basis for range selection. Range selection: choosing the quantization range from the observed distributions -- via min/max (the full range) or percentile (clipping the tails). Outliers: extreme values -- clip them (percentile -- better common-value precision, lose the outliers) vs preserve them (min/max -- cover the outliers, waste common-value precision) -- the key tradeoff.
Middle row: dimensions and methods. Static vs dynamic: static (precomputing the ranges from calibration data -- fixed at inference) vs dynamic (computing ranges at runtime per input -- adapting, but with runtime cost) -- the range-computation approach. Per-tensor vs per-channel: the granularity -- one range for a whole tensor (coarser) vs a range per channel (finer -- handling per-channel distribution differences -- more accurate, especially for weights) -- the range granularity. Methods: how the range is chosen -- min/max (simple), or optimizing a metric -- MSE (minimize the quantization mean-squared error), entropy/KL divergence (minimize the information loss) -- optimizing the range for minimal error. PTQ vs QAT: PTQ (post-training quantization -- quantizing a trained model using calibration data -- no retraining -- where calibration is key) vs QAT (quantization-aware training -- training with quantization, learning the ranges -- more accurate, but requires training).
Bottom rows: activations and validation. Activation calibration: activations are input-dependent (their ranges depend on the input -- unlike fixed weights) -- so their ranges must be calibrated on representative data (running the calibration data to observe the activation ranges) -- especially important (activations often have the outlier problems). Validation: measuring the accuracy loss (running the quantized model on a held-out set -- confirming the calibration preserved accuracy acceptably) -- the essential check. The ops strip: data selection (selecting the calibration data -- representative of the real inputs, enough of it -- the foundational requirement), method (choosing the calibration method -- min/max vs percentile vs MSE/entropy, per-tensor vs per-channel, static vs dynamic -- for the accuracy/cost balance), and validation (validating the quantized model's accuracy -- confirming the calibration worked, before deploying).
End-to-end flow
Trace calibrating activations for PTQ. A trained model is post-training quantized to INT8. The weights are fixed (their ranges can be computed directly from the weight values). But the activations are input-dependent -- so their ranges must be calibrated: the team runs a representative sample of inputs (the calibration data) through the model, observing the activation values at each layer (the actual distributions the activations take). From these observations, they select the activation quantization ranges -- using a percentile method (e.g., the 99.9th percentile -- clipping the extreme outliers) to balance covering the common values (with good precision) against the outliers (clipped) -- since the activations have outliers that would waste precision if fully covered. The chosen ranges (from the representative calibration data, percentile-clipped) quantize the activations well (good precision for the common values, outliers clipped) -- preserving accuracy. The calibration (representative data, observing activation distributions, percentile range selection) chose good activation ranges -- the key to accurate PTQ.
The outlier and granularity vignettes show the range choices. An outlier case: the team compares min/max vs percentile range selection. With min/max (covering the outliers), the common activation values are squeezed into few integer levels (the wide range wasting precision on the sparse outlier region) -- accuracy loss. With percentile (clipping the outlier tails), the common values get more precision (the range fitting them) at the cost of clipping the rare outliers -- and the overall accuracy is better (the common values, the majority, benefiting more than the rare outliers lose) -- so they use percentile. The outlier handling (percentile clipping) improved the accuracy. A granularity case: for the weights, the team uses per-channel quantization (a range per output channel -- since different channels have different value distributions) -- more accurate than per-tensor (one range for the whole weight tensor, which would mis-fit channels with different distributions) -- the finer granularity improving the weight quantization accuracy.
The method and validation vignettes complete it. A method case: the team uses an MSE-based method for range selection (choosing the range that minimizes the quantization mean-squared error -- rather than simple min/max) -- optimizing the range for minimal quantization error (better than naive min/max) -- the method improving the accuracy. A validation case: after calibrating and quantizing, the team validates the quantized model (measuring its accuracy on a held-out set -- confirming the accuracy loss is acceptable) -- and if the accuracy loss were too high, they'd revisit the calibration (more/better data, different method/granularity, or QAT) -- the validation confirming the calibration worked (or prompting revision). The consolidated discipline the team documents: calibrate quantization ranges using representative data (a representative sample of real inputs -- run through the model to observe the value distributions, especially input-dependent activations -- the foundational requirement), select ranges handling outliers (percentile clipping often better than min/max -- balancing common-value precision against outlier coverage), choose the granularity (per-channel more accurate than per-tensor, especially for weights) and method (MSE/entropy optimizing the range, better than naive min/max), use static or dynamic ranges per the accuracy/cost balance, calibrate activations carefully (input-dependent -- on representative data), validate the quantized model's accuracy (confirming the calibration worked), and consider QAT if PTQ calibration can't preserve accuracy -- because calibration largely determines whether quantization preserves accuracy or destroys it (choosing the ranges well from representative data, handling outliers), the key step in accurate post-training quantization.