Why architecture matters here
Attention decides how the model integrates context. If it costs too much memory, you cannot serve long contexts; if it costs too much compute, you cannot serve many users. A single variant rarely fits every layer: shallow layers benefit from cheap local windows while deep layers may need richer heads. That is why modern architectures treat attention as a per-layer decision rather than a one-size choice.
Getting this wrong shows up as long-context regressions, degraded recall, or the dreaded KV cache OOM at the exact tail of a session. Getting it right unlocks 4-16x throughput at the same quality — the difference between a demo and a viable product.
Architecture matters because the variant is not just math: it changes cache layout, sharding, kernel selection, and the eval you must run to prove nothing regressed. Treat attention as a system, not just a formula.
The architecture: every piece explained
The top strip anchors the trade-off: KV cache and quality vs context length. MHA baseline — H query heads and H KV heads — is the reference point; every other variant is a deviation. MQA collapses to a single KV head shared across all queries; cache shrinks by H×, but capacity for diverse retrieval drops. GQA is the middle ground: H queries but G groups of KV heads (G much less than H), preserving most quality at a fraction of the cache. Sliding window restricts attention to the last W tokens; compute drops from N² to N·W and cache to W per position. Linear attention replaces softmax with a kernel feature map, converting the cost to O(N·d²) — great for very long sequences, quality still trailing behind for many tasks.
The middle row is the machinery. KV cache is the per-layer per-head store that dominates memory at serve time — its shape changes drastically per variant. Flash kernels are IO-aware attention implementations that keep the softmax fused and blocked in SRAM; every variant benefits when a flash kernel exists. Attention sinks reserve the first K tokens as always-attended anchors for streaming windows so quality does not collapse at long horizons.
The lower rows are the control plane. Router at model config is a per-layer map: layer 0-5 use sliding window with sinks, layer 6-20 use GQA, layer 21-31 use MHA for the final integration. Serving budget translates this into an SLO: target 40 tokens/s per user at 200ms TTFT on this hardware. The eval harness at the bottom runs perplexity, long-context recall (needle in haystack), and throughput per candidate mix, so every choice is measurable.
End-to-end flow
End-to-end: a serving engineer opens the model config and toggles layer 0-5 from MHA to sliding-window-with-sinks. The KV cache footprint for those layers drops from N per token to W per token; long contexts become affordable. The router re-emits kernel selections, the flash-attention kernel is chosen for compatible variants. The serving budget dashboard recomputes: tokens/s rises from 22 to 41 on the same GPU. The eval harness runs 200 tasks — perplexity is flat, needle-in-haystack at 128k drops from 0.94 to 0.88, throughput is 1.9×. The engineer either accepts the quality trade or narrows the window from 4096 to 8192 tokens to recover recall. Every step is reversible, every step is measured.