Why architecture matters here
The architecture matters because the symmetric-versus-asymmetric decision is a direct trade between how much accuracy you keep and how fast the model runs, and getting it wrong costs you on whichever axis you neglected. Choose symmetric for a strongly one-sided tensor and you throw away half your representable levels, coarsening every value and degrading accuracy; choose asymmetric on the compute-critical weight path and you pay for cross terms in every matmul, giving back the speed quantization was supposed to buy. The choice is not cosmetic — it shows up as either a measurable accuracy drop or a measurable latency increase.
It matters because the two halves of a linear layer have genuinely different distributions, so a single global policy is wrong. Weights in a trained network are typically distributed roughly symmetrically around zero, so symmetric quantization fits them naturally and keeps the matmul cheap. Activations, by contrast, are often skewed — post-ReLU activations are non-negative, and many attention and normalization outputs are lopsided — so asymmetric quantization fits them far better. Understanding the mechanism is what lets you make the standard, correct split: symmetric weights, asymmetric activations, rather than forcing one scheme on both.
It matters because the zero-point interacts with the mathematics of the operation, not just the storage. A symmetric integer matmul is almost a normal integer matmul: multiply the integer weights and activations, accumulate, then scale once. An asymmetric matmul expands, because subtracting the zero-points before multiplying produces cross terms — a weight-times-activation-zero-point term and its mirror — that must be computed and folded in. Whether those correction terms can be precomputed and hoisted out of the inner loop, or must be evaluated per output, decides whether asymmetric quantization is nearly free or noticeably slower.
Finally, it matters because the scale and zero-point are where quantization error is born, and their choice interacts with everything else in the pipeline — the calibration that observes the range, the clipping that tames outliers, and the granularity (per-tensor versus per-channel versus per-group) at which you compute them. A symmetric scheme with a bad range is coarse; an asymmetric scheme with an outlier in the calibration set has its zero-point yanked toward the outlier, wasting levels on empty space. Seeing scale and zero-point as the foundational knobs clarifies why calibration, clipping, and granularity all exist: they are all in service of setting these two numbers well.
The architecture: every piece explained
Top row: from a float tensor to the two mapping parameters. Start with a float tensor and observe its actual range — the minimum and maximum values it takes on representative data. Pick the range to quantize: often not the raw min/max but a clipped range that ignores rare outliers, chosen by calibration so the bulk of the distribution is represented finely. Compute the scale as the width of that range divided by the number of integer levels available (255 for unsigned 8-bit, for instance). Then set the zero-point: fixed at 0 for symmetric, or an integer offset for asymmetric that aligns integer-zero with the low end of a one-sided range.
Middle row: the two quantization formulas and what they store. Symmetric quantizes by q = round(x / s) into a signed integer range centered on zero — integer 0 is exactly float 0, and the mapping is a pure scaling. Asymmetric quantizes by q = round(x / s) + zp, adding the zero-point so that the smallest float maps to the smallest integer and the whole observed range is covered. Either way you store the integers plus the scale (and, for asymmetric, the zero-point) — those are the packed quantized weights. At use time you dequantize with x' = s * (q - zp) to recover an approximate float (with zp=0 for symmetric, this is just s * q).
Bottom-left: why symmetric is cheap in a matmul. A symmetric matmul multiplies integer weights by integer activations and accumulates, and because both zero-points are zero there are no correction terms — the dequantization is a single scale applied to the accumulated integer sum. This is the property that makes symmetric quantization the default for the weight path: it preserves the simple, fast integer inner loop that hardware accelerates.
Bottom-right and ops: why asymmetric costs more, and how to choose. An asymmetric matmul must account for the nonzero zero-points: expanding (q_w - zp_w)(q_a - zp_a) yields the desired product plus cross terms involving each zero-point, which must be computed and folded in — some can be precomputed per weight row, but they are real extra work. The ops strip states the standard policy: choose per tensor, using symmetric for weights (centered, on the hot matmul path) and asymmetric for skewed activations (like the non-negative output of a ReLU), and tune granularity and clipping so the chosen scheme's scale and zero-point fit the real distribution.
End-to-end flow
Trace one quantized linear layer end to end: quantize its weights symmetrically, quantize a skewed activation asymmetrically, run the integer matmul, and dequantize the result.
Quantizing the weights (symmetric): the layer's weight matrix has values distributed roughly evenly around zero, from about -0.4 to +0.4. Symmetric quantization picks a range symmetric about zero (say [-0.42, +0.42]), computes a scale so that the signed 8-bit range spans it, and maps each weight with q = round(w / s). Integer 0 corresponds exactly to weight 0, no zero-point is stored, and the packed weights are just the integers plus one scale (or one scale per channel, at finer granularity). Because the weights are centered, no integer levels are wasted.
Quantizing the activation (asymmetric): the input to this layer is the output of a ReLU, so every value is non-negative, ranging from 0 to about 6. A symmetric scheme would reserve negative integer levels that this tensor never uses, halving its effective precision. Instead, asymmetric quantization sets the zero-point so that the smallest activation (0) maps to the smallest integer and 6 maps to the largest, spreading all 256 levels across the actual [0, 6] range. Each activation is quantized with q = round(a / s) + zp, and the scale and zero-point are stored alongside. The skewed tensor is now represented as finely as possible.
The integer matmul: the kernel multiplies the integer weights by the integer activations and accumulates in a wide integer accumulator. Because the weights are symmetric (zp=0), the only zero-point that appears is the activation's. The kernel expands the product to account for it: the main term is the integer dot product, and the correction term — the activation zero-point times the sum of the weight integers along each row — can be precomputed once per weight row and subtracted. This keeps the inner loop a clean integer multiply-accumulate, with the zero-point correction hoisted out, so asymmetric activations cost little on top of the symmetric weight path.
Dequantization and output: the accumulated integer result is converted back to float by applying the combined scale (weight scale times activation scale) and subtracting the precomputed correction, yielding the approximate float output of the layer, ready to feed the next operation. The whole layer ran in integer arithmetic on the hot path, with symmetric weights keeping the multiply cheap and asymmetric activations keeping precision high on the skewed input — the two schemes deployed exactly where each is strongest, which is the entire point of choosing per tensor.