StreamingLLM — Infinite Context via Attention Sinks

The workload that breaks the KV cache

Autoregressive decoding is fast only because of the KV cache: each new token attends to the stored keys and values of every previous token instead of recomputing them. That bargain is excellent for a 2k-token request and terrible for a stream that never ends. Cache size grows as 2 × layers × kv_heads × head_dim × tokens × bytes, strictly linear in length, and it is per-sequence.

Two costs compound. Capacity is the obvious one: cache crowds out weights and activations, the scheduler admits fewer sequences, and throughput falls even though the GPU is nominally busy. The quieter one is bandwidth. Decode is memory-bound; every step streams that sequence’s entire cache through the attention kernel, so a cache grown 10× makes each token roughly 10× more expensive. Latency degrades invisibly. Grouped-query attention and KV quantization shrink the constant factor, but nothing about them bounds the growth. Only eviction does.

Two obvious fixes, and why both fail

The first instinct is recompute with a window: throw the cache away and rebuild it from the most recent L tokens at every step. Quality holds up, because the model always sees a dense, correctly-positioned block of recent context. The cost is ruinous: you have converted a linear decode into a quadratic prefill per token.

The second instinct is a plain sliding-window cache: keep the last L keys and values, evict the oldest each step, never recompute. That is O(1) memory and O(L) work per token. It also falls apart. Perplexity is stable while the window is still filling, then explodes the moment the very first tokens are evicted — not gradually, but as a step change, often into fluent-sounding gibberish. The failure is too abrupt to be explained by ‘the model lost useful context.’ Something specific about those first tokens is holding the computation together.

The attention sink: where softmax dumps its leftover mass

The explanation is a property of softmax, not of language. Attention weights are normalized to sum to one, so every head must place its full probability mass somewhere on every token, even when the honest answer is “nothing here is relevant.” There is no abstain option.

Trained models converge on a consistent parking spot: the earliest tokens. Under causal masking, position 0 is the only key every later query can see, making it the one universally available target, and training bakes that in. Inspect a deep decoder’s attention maps and you find middle and upper layers dumping a large share of their mass onto the first handful of tokens regardless of what those tokens say — a BOS marker works as well as a meaningful word. These are attention sinks: semantically empty, structurally load-bearing.

Evict them and the mass has nowhere neutral to go. It is redistributed onto real tokens in the window, inflating their scores and pushing hidden states off the manifold the model was trained on. That is the step change.

The recipe: a few sinks plus a rolling window

Once the diagnosis is right, the fix is almost embarrassingly small. Keep the first few tokens in the cache permanently, keep a rolling window of the most recent tokens, and evict everything between. Roughly four sink tokens is the usual figure, enough to restore the sink with diminishing returns beyond that; the window is sized to the recent span the application needs and to your memory budget.

The result is a cache of fixed size n_sink + n_window that never grows again, however long the session runs. Perplexity stays flat over sequences far longer than the model’s trained context instead of diverging the instant the window rolls over. No retraining, no new kernels, no architecture change: a cache policy and a position fix.

Positions come from the cache, not from the transcript

The one implementation detail people get wrong is positional encoding. It is tempting to keep each token’s original index: sinks at 0..3, then a window at, say, 900,000..904,095. Do that and the scheme still breaks, because you are asking the model to reason about relative distances it has never seen.

Positions must be assigned by location in the cache, not location in the conversation. The sinks occupy slots 0..3, the window follows at 4, 5, 6, and the assignment shifts as the window rolls. With RoPE this means applying the rotation at attention time against cache-relative indices rather than caching pre-rotated keys; with relative-bias schemes the bias is computed on cache offsets. Every distance the model handles then stays inside its trained range, which is why the technique needs no context extension. The RoPE positional-extension family — position interpolation, NTK-aware scaling, YaRN — attacks the opposite problem: making the window itself longer.

What constant memory buys you on the GPU

From the serving stack’s point of view the attractive property is predictability. A sequence’s KV footprint is known at admission and never changes, so the scheduler can compute a hard concurrency ceiling and hold it. No creeping pressure, no preemption cascade at hour six, no long-session tenant starving the batch.

Implementation rides on the block-based paged KV cache machinery modern engines already have. The block table simply becomes a small ring: sink blocks are pinned and never recycled, window blocks are reused in rotation, and eviction is a pointer update rather than a copy or compaction pass. Nothing moves in HBM, and the attention kernel needs no change beyond gathering discontiguous blocks, which paged attention already does.

The payoff is the performance shape: per-token latency stops climbing with session age and flattens, because the bytes streamed per decode step are bounded by the window rather than the transcript.

Approximate, not exact — the FlashAttention contrast

This is the boundary that matters most, and it blurs easily because both get filed under ‘attention optimizations.’ They are not the same kind of thing.

FlashAttention is an exact method: it reorders the computation so the N×N score matrix is never materialized, and the numbers it produces are the numbers dense attention would have produced. Deploy it without thinking about quality. StreamingLLM is an approximation: tokens outside the sinks and the window are genuinely gone. The model stays fluent because the sinks preserve the distribution’s shape, but fluency is not recall. Ask about something said an hour ago and it cannot answer: that key never reaches the kernel. Stable perplexity describes local prediction quality, not long-range memory.

Trained sinks and the wider eviction family

If sinks work around softmax having no abstain option, the cleaner answer is to give it one. Pre-training with a dedicated learnable sink placeholder at the front of every sequence concentrates the behaviour into that single slot, after which streaming retains one token rather than four. That is a decision for whoever trains the model; for anyone serving an existing checkpoint, the first few real tokens are the sink you already have.

StreamingLLM also sits at the simple end of a family of KV eviction policies. Heavy-hitter and accumulated-attention schemes score tokens dynamically and keep whichever ones the model has been attending to, recovering more distant content at the price of bookkeeping on the decode hot path. StreamingLLM keeps a fixed positional pattern instead of a learned one: weaker in what it retains, and cheaper, with an eviction rule too trivial to become a bottleneck. For an open-ended stream that trade is usually right.

Streaming vs context extension vs retrieval

Three families address “the conversation is too long,” and they solve different halves of it. Context extension (position interpolation, NTK-aware scaling, YaRN, long-context fine-tuning) makes the usable window genuinely bigger, so the model can still attend to distant tokens. It costs attention over that larger window and a KV cache proportional to it, and however large you make it, it stays finite.

Retrieval keeps history outside the model and pulls relevant fragments back into the prompt on demand. It is the only one of the three that gives you long-term memory.

Streaming extends neither window nor memory. It buys survival: decoding coherently forever at fixed cost. The three compose cleanly — a long extended window, a rolling streaming policy over it, and retrieval to reinject the old facts that matter — and reaching for one when you needed another is the most common way teams get burned.

When to reach for it

Streaming with sinks fits genuinely unbounded, locally-coherent workloads: an always-on assistant, a log or telemetry narrator, live transcription and summarization, a monitoring agent running since last quarter. What these share is that the next token depends on recent context, and old context is legitimately disposable.

It is the wrong tool whenever correctness depends on the whole input. Document question-answering, code understanding across a repository, multi-hop reasoning over a long transcript, summarizing a full session: in all of these, dropping the middle drops the answer, and the model produces something confident and wrong rather than admitting the gap. Because the failure is silent, evaluate on long-range retrieval tasks, not perplexity, before shipping. Two adjustments make it safer: size the window to a real unit of work, and pair it with retrieval so anything important can come back into the window instead of trusting a cache that was always going to forget it.