Why architecture matters here
The architecture matters because the scale factor is where quantization succeeds or fails, and the two timing choices fail in different, predictable ways. A scale that is too tight clips large activations to the integer maximum and destroys information; a scale that is too loose wastes precision on a range the data never uses, coarsening every value. Weights are easy — they are fixed after training, so you can measure their exact range once and pick optimal per-channel scales offline. Activations are the hard part, because they depend on the input: the range of values flowing through a layer for one prompt can differ wildly from another. Whether you freeze an activation scale from calibration data or recompute it live is therefore the central design decision, and it directly determines robustness to inputs you did not anticipate.
The second forcing function is latency, and here static and dynamic diverge sharply. Static quantization's baked scales let the compiler fuse the entire integer pipeline — quantize input, INT8 matmul, requantize output — into tight kernels with no data-dependent branches, extracting the full throughput win of integer hardware. Dynamic quantization must, on every call, scan each activation tensor to find its range and derive a scale before it can quantize, and that reduction is pure overhead that cannot be precomputed. For large matmuls the INT8 compute still dominates and the overhead is small; for small tensors or bandwidth-bound layers the runtime range calculation can eat much of the speedup, which is why the choice is workload-dependent, not universal.
The third reason is operational: static quantization introduces a calibration artifact and a distribution-shift risk that dynamic quantization simply does not have. A statically quantized model carries scales derived from a specific dataset at a specific time; if production data drifts — new languages, new input patterns, a seasonal shift — the frozen scales may no longer fit, and accuracy degrades silently. Dynamic quantization sidesteps this entirely by always fitting the current data, which is why it is the safer default for models with volatile or unpredictable activation distributions, and why LLM inference on the CPU often defaults to dynamic quantization of the linear layers.
The architecture: every piece explained
Top row: the shared start and the fork. The FP model is the trained network in full precision. Weight quantization is identical for both paths and happens offline: because weights are static, their exact range is known, so each channel gets an optimal scale and the weights are stored as INT8 once. The paths diverge on activations. The static path runs a calibration set through the model to observe activation ranges and derives a fixed scale per activation tensor, baked into the model. The dynamic path stores no activation scales at all; instead it computes them per tensor, per call, from the live data at runtime.
Middle row: the machinery of each path. Calibration (static) feeds representative inputs through the FP model and records the distribution of each activation — typically as a min/max or a percentile to reject outliers — which the scale store holds as the frozen activation scales. Runtime range calculation (dynamic) is the per-call reduction that scans an activation tensor to find its min/max (or absolute max) and derives the scale on the spot, immediately before quantizing. Both paths then converge on the INT matmul kernel: the INT8 inputs and INT8 weights multiply-accumulate in a wide integer accumulator, and the result is requantized back to the output tensor's scale (or dequantized to FP for the next layer).
Bottom rows: the decision criteria and control. The accuracy check evaluates the quantized model against the FP baseline on a real task; static quantization must clear this bar with its calibrated scales, and a failure points to bad calibration or unhandled outliers. The latency budget weighs static's fused, zero-overhead kernels against dynamic's per-call range computation, and this budget is what ultimately decides the path for each layer — you can, and often should, mix them. The ops strip names the signals that keep a quantized deployment healthy: measured accuracy drop versus FP, throughput, outlier or distribution drift that would invalidate static scales, and the per-layer precision choices in effect.
End-to-end flow
Walk a linear layer through both paths for the same input. In the static deployment, the activation arriving at the layer is quantized to INT8 using the scale that calibration froze for this tensor — no range computation happens, the scale is just read from the model. The INT8 activation and the pre-quantized INT8 weights go straight into the fused integer matmul, which accumulates in INT32 and requantizes the output using the pre-derived output scale. The entire path is branch-free and precomputed; the layer runs at full integer-hardware throughput. This is fast and cheap — provided the incoming activation actually falls within the range calibration observed.
In the dynamic deployment, the same activation first triggers a runtime reduction: the kernel scans the tensor, finds its actual min and max for this specific input, and derives a scale that fits this data exactly. Only then is the activation quantized and fed into the INT8 matmul. The scale perfectly matches the current data, so there is no clipping and no wasted range — but the scan cost was paid on this call and will be paid again on the next. For a large matmul the scan is a small fraction of the matmul time; for a tiny layer it may rival it.
Now inject a distribution shift: a burst of inputs produces activations far larger than anything in the calibration set. The static path clips those large values to its frozen integer maximum — the baked scale simply cannot represent them — and accuracy on that burst degrades, silently, with no signal beyond a downstream metric dip. The dynamic path handles the same burst gracefully: its runtime range calculation sees the large values and widens the scale to fit them, so nothing clips. This is the crux of the trade made concrete — static is faster but brittle to unseen distributions; dynamic is slightly slower but robust to them.
The mature answer is rarely all-or-nothing. A production model profiles each layer: linear layers with volatile activations and generous matmul sizes go dynamic (robustness cheaply bought), while stable, latency-critical layers where calibration is trustworthy go static (maximum throughput). The accuracy check validates the mixed configuration against FP, the latency budget confirms the static layers earn their fused speed, and monitoring watches for the outlier drift that would tell you a static layer should be reconsidered. The architecture is thus not a single choice but a per-layer assignment guided by two measurements: how much accuracy each path costs and how much latency each path saves.
It is worth naming a middle ground that the two paths converge toward in practice. A static deployment can be made far more robust by recalibrating on a schedule from recent production traffic — freezing scales, but freezing them from data that reflects the present rather than launch day — which recovers much of dynamic's robustness while keeping the fused-kernel speed. A dynamic deployment, conversely, can amortize its runtime cost by computing the range once per larger unit of work rather than per tiny tensor, or by caching a recent range and only recomputing when it drifts. The clean static-versus-dynamic dichotomy is thus the starting frame; real systems tune along the continuum between 'scale computed once, long ago' and 'scale computed fresh, every call', placing each layer where its accuracy sensitivity and latency budget point.