Why architecture matters here
The economics of inference are dominated by memory, not arithmetic. For a large language model serving tokens, the bottleneck is moving billions of weight parameters (and a growing KV cache) between high-bandwidth memory and the compute units; the matrix multiplies themselves are comparatively cheap. Cutting the weights from sixteen bits to four quarters the bytes moved per token, which nearly quarters the memory-bandwidth cost and lets a given accelerator hold a much larger model or serve many more concurrent requests. Four-bit quantization is one of the highest-leverage optimizations available — if it can be done without wrecking accuracy.
What wrecks accuracy is outliers. Empirically, transformer activations develop a small number of channels whose magnitudes are enormous relative to the rest — a systematic, structured phenomenon tied to how attention and layer normalization interact, not random noise. A uniform quantizer maps a range of real values onto a small set of integer levels; with four bits there are only sixteen levels to cover the whole range. If one channel's values reach 100 while the rest live in [-2, 2], the quantizer must stretch its sixteen levels across [-100, 100], giving the ordinary values a resolution so coarse that meaningful distinctions vanish. The rare outlier is quantized fine; everything else is destroyed.
The insight of rotation-based methods is that outliers are a basis-dependent phenomenon. A value being an outlier means it is large in the coordinate system the network happens to use — but that coordinate system is not sacred. If you rotate the representation into a different orthonormal basis, the same information is preserved (a rotation loses nothing) but the energy that was concentrated in a few coordinates gets redistributed across all of them. A well-chosen rotation turns a spiky, outlier-dominated vector into a smooth one whose values are all of comparable magnitude — exactly the distribution a low-bit quantizer handles well. The outliers do not have to be isolated or scaled; they can be spread.
The reason this is practical and not just pretty is that transformers are full of linear layers, and a rotation is a linear operation that commutes cleanly with them. Because an orthogonal matrix and its transpose multiply to the identity, a rotation applied before a linear layer can be undone by folding its inverse into that layer's weights, at no runtime cost and with no change to the computed result. The network is computationally invariant to these rotations, so they can be inserted purely to improve quantizability, making the whole scheme nearly free and, crucially, requiring little or no calibration data.
The architecture: every piece explained
Walk the diagram. On the left, the weights and activations arrive with their characteristic outlier channels — a few coordinates carrying most of the range. The rotation matrix Q is an orthogonal matrix chosen to spread that energy; in practice a randomized Hadamard matrix is used because it is orthogonal, mixes all channels maximally, and — critically — can be applied in n log n time via a fast Hadamard transform rather than a full matrix multiply. Applying Q produces rotated tensors in which the outliers are smeared across channels and no single coordinate dominates, so the value distribution is now friendly to a four-bit grid.
The key implementation move is folding the rotation into the weights offline. For a linear layer computing y = Wx, inserting a rotation means computing y = (WQ)(Qᵀx) — the input is rotated by Q-transpose, and the weight matrix is replaced by WQ. But WQ can be computed once, ahead of time, and stored as the new weights; at runtime there is no extra cost on the weight side at all. The rotation is 'baked in.' Where a rotation must be applied to a runtime activation that cannot be pre-folded, the online Hadamard transform does it cheaply — its n log n cost is negligible next to the matrix multiplies it enables to run in four bits.
The low-bit quantization now operates on rotated tensors whose ranges are tight and outlier-free, so both weights and activations quantize to INT4 with small error. This is the payoff: the same INT4 quantizer that was catastrophic on the raw tensors is nearly lossless on the rotated ones, because the rotation did the hard work of reshaping the distribution.
The invariance box is the correctness guarantee. Because Q is orthogonal, QᵀQ = I: wherever a rotation is applied, a matching inverse rotation appears downstream (folded into the next weight matrix or applied as its own Hadamard), and the two cancel. The network's output is mathematically identical to the unrotated model up to quantization error — the rotations change the basis in which computation happens, never the result. The KV-cache rotation box extends the idea to the attention cache: the keys and values stored per token are themselves rotated before being quantized, so the memory-hungry KV cache — which grows with context length and often dominates memory at long contexts — can also be held in four bits without the outliers in key/value channels destroying attention accuracy. The bottom ops strip captures what shipping this requires: it is calibration-light (rotations are largely data-free), it needs kernel support for fused Hadamard-plus-INT4-matmul, and it demands rigorous accuracy validation.
End-to-end flow
Trace a single transformer block through a rotation-based four-bit quantization, from the offline preparation to a runtime forward pass. Take the attention projection and the subsequent feed-forward layer.
Offline: choose and fold the rotations. A randomized Hadamard matrix Q is selected for the model's hidden dimension. For each linear layer y = Wx where the input carries outliers, the tooling computes the folded weight WQ and stores it; conceptually the layer now expects a Q-transpose-rotated input and produces its output already compensated. Rotations are inserted in matched pairs across the residual stream so that every forward rotation has a cancelling inverse folded into a downstream weight. Because this only requires the weight matrices — not a dataset — the process is essentially calibration-free; at most a small sample is used to pick quantization scales, not to learn the rotations.
Offline: quantize. With the rotations folded in, the folded weights WQ are quantized to INT4. Because the rotation was chosen to make these matrices outlier-free, the four-bit representation captures them with small error — the same quantizer that would have been ruinous on the raw W is now accurate. The model ships as INT4 weights plus a specification of which activations need an online Hadamard at runtime.
Runtime: the forward pass. A token's activation enters the block. Where the design calls for it, an online Hadamard transform rotates the activation — an n log n operation, cheap. The rotated activation is quantized to INT4 and multiplied against the INT4 folded weights in a fused low-bit kernel; the bulk of the block's compute now runs in four bits, at a fraction of the memory traffic of the sixteen-bit original. As the data flows to the next layer, the inverse rotation that was folded into that layer's weights cancels the forward rotation, so the values entering the next operation are — up to quantization error — exactly what an unrotated, unquantized model would have produced.
The KV cache. When this block writes keys and values into the attention cache, those tensors are rotated and quantized to four bits too, so a long-context request whose cache would otherwise dominate memory now stores it compactly. On a later token, attention reads the four-bit rotated cache, and the rotation invariance ensures the attention scores match the full-precision computation closely. The net result: a model that runs in four bits end to end — weights, activations, and cache — with accuracy close to the original, achieved by spinning the outliers away rather than tiptoeing around them.