Calibration samples and held-out error

Configure sample calibration

Compare reconstruction

Scalar errors on the listed samples, not model-quality measurements. Scroll tables horizontally on narrow screens.

Inspect calibration samples

Inspect held-out samples

Calibrate samples, then inspect separate data

This experiment takes explicit scalar activation samples and computes a static quantizer from the calibration list. The held-out list is reconstructed using that same quantizer. It never contributes to the fitted range. These small arrays are teaching fixtures, not activations collected from a trained language model. You can edit both lists to inspect distribution mismatch without assigning invented ranges to English, code or multilingual datasets. The displayed errors measure scalar reconstruction only; they do not measure task accuracy or language quality.

Choose an example and edit the evidence

Example selects a named fixture; Load example copies its two arrays into the editors while preserving bit width and the range method. Matched range contains calibration values from minus one to one and held-out values inside that interval. Shifted evaluation keeps that calibration range but includes held-out values at minus three and three. Rare calibration outlier adds twenty to an otherwise small calibration list. All-zero calibration exposes the degenerate range explicitly. Each editor accepts a JSON array of one through one hundred twenty-eight finite numbers, with magnitudes at most one thousand. Edits recalculate immediately. Invalid input clears all tables instead of leaving old measurements visible.

Define the range selection exactly

Absolute maximum selects the largest absolute calibration value, giving a symmetric interval even when the samples are one-sided. Nearest-rank percentile first sorts absolute calibration values, then chooses the value at rank ceil(p divided by one hundred times sample count), using one-based ranks. Percentile is enabled only for that method and accepts values from one through one hundred. At one hundred it equals the absolute maximum. This exact order-statistic rule uses no histogram or interpolation and is not claimed to reproduce the percentile implementation of a particular runtime. Complete state evidence exposes the sorted magnitudes and chosen rank.

Understand bits, scale and rounding

Bit width selects two, three, four or eight bits. With b bits, the maximum integer code Q is two raised to b minus one, minus one. The symmetric codes run from minus Q through Q, leaving the most negative signed integer unused. The scale is the chosen clip magnitude divided by Q. Each input is clipped to the fitted interval, divided by scale, rounded to the nearest integer with ties to the even integer, and multiplied by scale. Clipped means the original magnitude exceeded the fitted interval; it is distinct from rounding to an endpoint. The tables show the bounded value, code, reconstruction and signed reconstruction-minus-input error.

Reproduce an outlier tradeoff

Reset defaults, select Rare calibration outlier and press Load example. Keep four bits and Absolute maximum: the clip magnitude is twenty, the scale is twenty sevenths, and all six held-out values round to zero. Switch to Nearest-rank percentile with percentile ninety. There are ten calibration samples, so rank nine selects magnitude one. The calibration value twenty is now clipped, but the smaller grid reconstructs the six held-out values more accurately. Compare both MSE rows: sacrificing the rare calibration outlier improves this particular held-out set while worsening its own reconstruction. It is not a promise that percentile calibration improves every evaluation distribution.

Change evaluation without leaking it into calibration

Keep the outlier experiment at percentile ninety and replace held-out samples with [-3,0,3]. The clip magnitude must stay one, both outer values clip, and held-out mean squared error becomes eight thirds. Increasing bit width cannot restore values outside that interval: the two endpoints still reconstruct as minus one and one. Change only percentile to one hundred to include the calibration outlier again, then compare rounding error against clipping error. Evaluation edits never change the selected range. Choosing settings after repeatedly inspecting held-out results nevertheless makes that list a tuning set; a real assessment needs a further untouched test set.

Handle zero ranges and exact ties

With All-zero calibration, the selected magnitude is zero. Every reconstruction is explicitly zero; the displayed scale one is a sentinel preventing division by zero, not a range learned from data. Nonzero held-out inputs are marked clipped and still contribute their full squared error. To inspect nearest-even ties instead, enter [-7,7] as calibration and [-2.5,-1.5,-0.5,0.5,1.5,2.5] as held-out data, choose Absolute maximum and four bits. Scale is one and integer codes are -2, -2, 0, 0, 2, 2. Negative and positive halfway cases follow the same even-integer rule.

Read measurements within their limits

MSE averages squared reconstruction errors across the listed samples; MAE averages absolute errors, and maximum error reports the largest absolute error. Clipped and zero-code counts are sample counts, not percentages. The calibration and held-out lists may have different lengths and are summarized independently. Reset defaults restores the matched fixture, four bits, absolute maximum and percentile ninety. Tables scroll horizontally on narrow screens and can receive keyboard focus; Complete state evidence expands the full precision calculation. This direct numerical tool has no artificial playback clock. Division uses the rounded JavaScript scale, so a mathematical halfway value can lie slightly off a tie in machine arithmetic. A positive range too small to represent its scale is rejected explicitly. It uses JavaScript number arithmetic, one shared scalar range, no zero-point offset, no per-channel scales, no activation collection pipeline, and no weight or matrix-output quantization. Dataset size alone cannot establish representativeness.

Production context: ONNX Runtime static quantization and calibration. Its runtime implementations differ from this explicit teaching rule.