Why architecture matters here
To see why rotation is the right tool, you have to see why transformers are uniquely hostile to naive low-bit quantization. Quantizing a tensor to INT4 means mapping its real values onto sixteen levels spanning from its minimum to its maximum. If the values are roughly uniform, sixteen levels resolve them fine. But transformer activations contain outlier channels: a few feature dimensions whose magnitudes are ten or a hundred times larger than the rest, and they appear consistently in the same channels. Those outliers set the max, so the sixteen levels are stretched across an enormous range, and the ordinary values — which carry most of the information — get crushed into just two or three levels. The result is catastrophic accuracy loss even though the vast majority of numbers are perfectly ordinary.
Prior methods attacked this in various ways — keeping outliers in higher precision (mixed precision), scaling outlier channels down and compensating in the weights (SmoothQuant), or clipping. Rotation is a different and more complete idea: instead of isolating or taming the outliers, eliminate the basis in which they exist. An outlier is a property of the coordinate system — the fact that energy is concentrated in specific channels. Rotate the coordinate system and that concentration is spread out: a vector that was 'huge in channel 7, small everywhere else' becomes 'moderate in every channel' after a rotation that mixes the channels. The peak-to-average ratio, which is what kills quantization, drops dramatically.
The reason this is mathematically free is orthogonal invariance. A transformer's core operations are matrix multiplications, and for an orthogonal matrix R (one where R R^T = I), you can insert R and R^T around a matmul without changing its result: x W = (x R)(R^T W). So if you rotate the activations by R and the weights by R^T, the full-precision output is byte-for-byte the same. Rotation is a change of basis that the network cannot distinguish — until you quantize, at which point the rotated basis quantizes far better because it has no outliers. You get outlier removal with exactly zero change to the underlying function in full precision.
SpinQuant's specific advance is that not all rotations are equal, and the best one can be learned. A random orthogonal rotation (or a fixed Hadamard) already helps by spreading outliers, but the amount of residual quantization error depends on the exact rotation chosen. SpinQuant treats the rotation matrices as parameters and optimizes them on a small calibration set to minimize the end-to-end quantization loss, subject to staying orthogonal (via optimization on the Stiefel manifold or a Cayley parameterization). Learning the rotation squeezes out the last accuracy gap that a generic rotation leaves on the table, which is often the difference between INT4 being usable and not.
The architectural consequence is that quantization becomes a two-part problem: where to place rotations in the network so they compose correctly with attention and the feed-forward blocks, and which rotations to use. Placement is a structural decision baked into the model graph; the rotation values are learned. Getting placement right is what lets some rotations be merged into weights offline (costing nothing at runtime) while only the unavoidable ones run online — and that split between merged and online rotations is the whole reason SpinQuant is fast enough to deploy rather than a theoretical curiosity.
The architecture: every piece explained
Follow the pieces. The input is an ordinary FP16 model with outlier-heavy activation channels. SpinQuant introduces a set of rotation matrices — conventionally labeled R1 through R4 — placed at specific points in the transformer: around the residual stream, inside attention (on the query/key/value and output projections), and in the feed-forward block. Each is orthogonal, and each is positioned so that its inverse appears on the other side of a matmul, preserving the computation. The placement is what makes the rotations composable with the architecture rather than a blunt transform bolted on the outside.
Rotating the weights happens offline and, crucially, is often free at inference. Where a rotation sits adjacent to a weight matrix, you can multiply it into that matrix once, ahead of time, producing a rotated weight W' = R^T W R that you then quantize and ship. At runtime the model just uses W'; the rotation has been absorbed and costs nothing. This is the merged rotations path, and maximizing how many rotations can be merged is a key design goal because merged rotations are the free ones.
Rotating the activations is the part that cannot always be merged, because activations are computed at runtime and a rotation on the residual stream or between attention heads has to be applied online during the forward pass. Done naively this is a full matrix multiply per rotation — expensive. SpinQuant makes it cheap with the Hadamard fast path: a Hadamard matrix is a special orthogonal matrix whose transform can be computed in O(n log n) time via a fast Walsh-Hadamard transform, like an FFT, instead of O(n^2). So the online rotations that must run in the forward pass use Hadamard structure and add only a small overhead relative to the matmuls they sit beside.
The calibration set and the learning loop are where SpinQuant differs from fixed-rotation methods. A few hundred representative sequences are run through the model, and the rotation matrices are optimized to minimize the difference between the quantized model's outputs and the full-precision outputs, while a Cayley or Stiefel-manifold parameterization keeps them orthogonal throughout optimization. The learned rotations are then frozen. The quantizer takes the rotated weights and activations — now outlier-free — and maps them to INT4 (weights) and INT8 (activations) with ordinary per-channel or per-group scales, which now work well because the rotated distributions are well-behaved.
Finally, fused kernels make the whole thing performant. An online Hadamard rotation followed by a quantized matmul is fused into a single kernel so the rotated activations never have to be written to memory and read back — the rotation happens in registers immediately before the low-bit multiply. The result is a model that stores its weights at roughly a quarter of FP16 size, runs on INT4/INT8 tensor-core paths, and — because the outliers were rotated away rather than clipped or approximated — retains accuracy remarkably close to the original, with only the small online-rotation overhead as runtime cost.
End-to-end flow
Walk the offline preparation first. You take the trained FP16 model, decide the rotation placement (R1 on the residual stream, R2 inside attention, R3/R4 in the feed-forward and value paths), and run the calibration set through it. The optimizer adjusts the rotation matrices to minimize the quantized-vs-full-precision output gap, keeping each matrix orthogonal. This is a modest optimization — a few hundred calibration samples, a handful of small matrices — not a retraining of the model, so it costs minutes to hours, not the days full fine-tuning would.
With the rotations learned, you fold in what you can. Rotations adjacent to weights are merged: each such weight matrix is replaced by its rotated form R^T W R and then quantized to INT4. These merged rotations vanish into the weights — at inference there is no separate rotation step for them at all. The rotations that remain are the online ones on the activation path; those are configured to use Hadamard matrices so the runtime can apply them with the fast transform.
Now run a token through the deployed model. Activations flow into a block and hit an online rotation point. The fused kernel applies the Hadamard rotation to the activation vector in O(n log n) — spreading whatever outlier energy is present evenly across channels — quantizes the rotated activation to INT8, and multiplies it against the INT4 rotated weights on the tensor cores. Because both operands were rotated into an outlier-free basis before quantization, the low-bit multiply loses very little precision, and because the rotation and matmul are fused, the rotated activation never round-trips through memory.
Contrast this with the same token through a naively quantized model. At the outlier channel, the INT4/INT8 quantization grid — stretched to cover the outlier's huge magnitude — represents the ordinary values with almost no resolution, so the matmul accumulates large rounding errors that compound layer after layer, and the model's output drifts badly from the FP16 reference. The visible symptom is a big jump in perplexity and broken downstream task accuracy. SpinQuant's rotated path never encounters that stretched grid because there is no outlier channel to stretch it — the rotation removed the very feature that made naive quantization fail.
The end state, verified on the calibration and held-out data, is a model whose INT4 outputs track its FP16 outputs closely: perplexity within a small margin of full precision, downstream accuracy largely preserved, memory footprint quartered, and throughput improved by the low-bit kernels — minus the modest, bounded overhead of the online Hadamard rotations, which the fused kernels keep small. The learned-rotation step is what pushes the accuracy from 'usable' (as generic rotation gives) to 'near-lossless,' and the merge-what-you-can discipline is what keeps the runtime cost low enough to ship.