Ring attention answers a narrow question with a very specific trick: how do you run exact attention over a sequence that no single GPU can hold? The answer is to shard the sequence itself across devices, keep each device’s queries pinned in place, and pass the key/value blocks around a ring of peers step by step. After one full lap every query has been scored against every key, and no device ever stored more than its own slice plus one block in flight. The mathematics is unchanged — this is exact attention, not an approximation — but the memory ceiling on context length is now set by the ring, not by one card’s HBM. This piece walks the mechanics: the rotation, the merge that makes partial results combinable, the overlap that hides the transfer, the causal-mask imbalance that ruins a naive implementation, and the regime where the ring stops paying for itself.
The ceiling ring attention is built to lift
Attention has two distinct memory problems, and it helps to keep them apart. The first is the N×N score matrix, quadratic in sequence length. That one is already solved on a single device by tiled, IO-aware kernels that never materialize the matrix — see FlashAttention for the SRAM-residency argument.
The second problem survives that fix. Even with a perfect fused kernel, the device still has to hold the keys, values, queries and activations for the tokens it is processing. Those grow linearly in N, and linear growth against a fixed HBM budget still terminates. At long enough context the per-layer K and V tensors alone exceed what one card can store, and no kernel cleverness helps, because those bytes are genuinely required rather than intermediate. Ring attention attacks precisely that second wall: it spreads the linear-in-N state over P devices so the ceiling moves with the device count.
Sharding the sequence, pinning the queries
The layout is the whole design. A sequence of N tokens is cut into P slices and device r receives slice r: its queries Q_r, its keys K_r and its values V_r, each of length N/P. Every other tensor in the layer — weights, the MLP, layer norms — is replicated or sharded exactly as it would be without ring attention, since only attention needs cross-token information.
The asymmetry that makes the scheme work: queries never move, KV does. Each device keeps Q_r resident for the whole operator and computes partial attention against whatever KV block currently sits in its buffer. Moving KV rather than Q is the right choice because the output is indexed by query — a travelling query would have to carry its accumulator with it and eventually come home, while a travelling KV block is read-only and is discarded the moment it has been consumed.
One lap around the ring
The devices form a logical ring: r sends to r+1 and receives from r−1, modulo P. The operator runs exactly P steps. At step 0 each device attends Q_r against its own K_r, V_r, then ships that KV block to its successor, receives one from its predecessor, and repeats. After P steps every block has visited every device once, so every query has been scored against every key.
Two properties fall out of this. The total FLOP count is unchanged — the same N² score-and-weight work, merely partitioned. And the communication volume is fixed and predictable: the bytes crossing any single link per step are just one KV block. Nothing broadcasts, nothing all-gathers, and no device ever holds more than two KV blocks at once.
Why partial results are allowed to merge
The step that makes the whole thing legal is the merge. Softmax normalizes over the entire key axis, so a partial output computed against one KV block is not a fraction of the answer — it carries the wrong denominator. Naively summing partial outputs is wrong.
The fix is the online-softmax identity, the same one that lets a single-device kernel stream tiles; the derivation lives in FlashAttention and is not repeated here. Operationally, each device carries three running quantities per query row: the maximum logit seen so far, the running sum of exponentials, and the running weighted output. When a new block arrives, its local max and sum are computed, both accumulators are rescaled by the exponential of the difference between the old and new running maxima, and the block’s contribution is added. The rescale is a cheap elementwise pass. Because the correction is exact, the result after P merges is exact attention — not a windowed or sparse variant.
Hiding the ring behind the GEMMs
A ring step that sends a KV block and then waits for it costs P serialized transfers, which makes long context slow rather than merely possible. Real implementations therefore double-buffer: the send of block i and the receive of block i+1 are posted as non-blocking peer-to-peer operations on a separate stream, and the attention kernel for block i runs concurrently against the buffer it already owns. The device stalls only if the transfer has not landed when the kernel finishes.
Whether it lands in time is a straightforward ratio. The compute per step scales with local queries times block keys times head dimension; the traffic is one KV block — two tensors of N/P tokens by head dimension by head count, at whatever precision the cache uses. Longer local slices raise compute quadratically against a linear rise in bytes, which is why ring attention hides its communication well at large context and badly at small.
The causal mask wrecks the load balance
Everything above assumes each of the P steps costs the same. Under a causal mask with contiguous slices, it emphatically does not. Device 0 holds the first tokens of the sequence, so its queries may only attend to keys from its own slice — every other block that arrives is fully masked and its work is thrown away. The last device holds the final tokens, whose queries attend to every block in the sequence, so none of its work is masked.
The result is a linear ramp: rank 0 does roughly one block of useful work, the last rank does P. Because the ring is synchronous, every step runs at the speed of the slowest participant, so the operator advances at the last rank’s pace while earlier ranks idle. A naive causal ring wastes a large share of its aggregate GPU time on masked tiles and waiting — an imbalance that reads as a fabric problem in a profiler but is really a partitioning problem.
Striping the sequence to flatten the triangle
The fix is to stop giving each device a contiguous run of the sequence. Cut the sequence into 2P chunks instead of P and give device r both chunk r and chunk 2P−1−r. Device 0 now owns the very first chunk and the very last one; the last device owns two middle chunks. Every rank gets one early, cheap chunk and one late, expensive one, and the triangular work is shared almost evenly. This paired or “zigzag” assignment costs nothing at runtime — it is purely a change in which tokens each rank was handed.
The bookkeeping cost is modest: the kernel must know the global position of every local token to build the mask, so position indices travel with the shard and each step’s mask becomes a function of the local chunk ids and the arriving block’s. Wholly masked blocks can then be skipped outright rather than computed and discarded.
What memory actually does as the ring grows
Per device, the attention operator now holds its own Q_r, K_r, V_r at N/P tokens, the running softmax accumulators over the local queries, and two KV block buffers — the one being consumed and the one arriving. Every one of those terms is N/P or smaller. Doubling the ring halves the per-device attention state, so the reachable context grows roughly in proportion to the number of devices you are willing to spend.
Two caveats keep this honest. Model weights are not sharded by the ring; they are replicated across it unless you also apply tensor or ZeRO-style sharding, so weights set a floor the ring cannot lower. And activations outside attention still scale with the local slice. The ring lifts the attention ceiling specifically; it does not make a model fit that never fit.
When the ring stops paying for itself
Ring attention is a bandwidth bet, and it loses in identifiable regimes. If the per-device slice is short, compute per step is small and the KV transfer is exposed rather than hidden — splitting a modest sequence over many ranks makes the operator slower, not faster. Slow links do the same at longer sequences: a ring closed over NVLink inside one node tolerates far smaller slices than one closed over InfiniBand across nodes, and a ring crossing both must be sized for its slowest hop.
Autoregressive decoding is the sharpest failure case: with a single query token per step there is essentially no compute to hide behind the rotation, so sharding the resident KV cache some other way is usually better. Grouped-query and multi-query attention cut KV bytes substantially, which helps the ratio — but they shrink the traffic, not the structure of the trade.
Where the ring sits in the parallelism stack
Ring attention is orthogonal to the other axes, which is why it composes rather than competes. Tensor parallelism splits heads and hidden dimensions within a layer; pipeline parallelism splits layers across stages; data parallelism replicates the model over batches. Ring attention splits the token axis, and large jobs run all four at once — usually with the ring closed over the fastest links, since it communicates every layer.
The practical reading: reach for it when context length, not model size, is what breaks you, when the per-device slice stays long enough to hide a KV hop, and when the causal partitioning is already fixed. Under those conditions it delivers exact attention at a context no single device could host, for a communication cost a fast fabric largely absorbs. Outside them, a cheaper axis of parallelism is the better answer.