Why architecture matters here
The architecture matters because the KV cache is the binding constraint on modern LLM serving economics, and MLA attacks it at the information level rather than the head-count level. Consider the arithmetic: for standard multi-head attention, the cache stores, per token per layer, a key and a value of dimension (num_heads × head_dim) each. Multiply by layers, by sequence length, by batch size, and by two bytes per element, and a long-context batch runs into tens or hundreds of gigabytes — often more than the model weights themselves. Because that memory is what limits concurrency and context length, halving or quartering it directly multiplies the requests a given cluster can serve and the context it can offer. GQA already exploits this by sharing KV heads; MLA pushes further by storing a single compressed latent whose dimension can be made much smaller than even a grouped set of KV heads, so its cache footprint per token can be a small multiple of a single head's size while still feeding all the query heads.
It matters, too, because MLA is a genuinely different point on the quality/memory curve rather than a more aggressive version of the same trade. MQA and GQA reduce the number of distinct KV projections, so different query heads increasingly attend through the same key/value subspace — the heads' independence, which is the entire reason multi-head attention works, erodes as you share more. MLA keeps per-head keys and values (each head still reconstructs its own K and V from the latent through its own slice of the up-projection), so the heads retain distinct views; what is shared is a low-rank bottleneck that the model is trained to pack efficiently. Empirically this lets MLA match or beat GQA's quality at a smaller cache, which is why it anchors the attention design of frontier open models rather than sitting in a paper as a curiosity.
The architecture matters because of a subtlety that would otherwise sink the whole idea: rotary position embeddings (RoPE) are applied to keys and queries as a position-dependent rotation, and that rotation does not commute with an arbitrary low-rank up-projection. If you compress a RoPE'd key into a latent and reconstruct it, the position information is mangled. MLA's answer — a decoupled RoPE path where a small dedicated key dimension carries position outside the compressed latent — is not a detail but the load-bearing design decision that lets low-rank KV compression coexist with rotary positions at all. Understanding it is the difference between reimplementing MLA correctly and producing a model whose long-context behavior quietly falls apart.
Finally it matters because the weight-absorption trick changes the inference cost story entirely. Naively, reconstructing full-size keys and values from the latent every step would add compute that partly offsets the memory savings. The observation that the up-projection can be mathematically absorbed into the query and output weight matrices means the model attends directly against the cached latent — smaller cache and no reconstruction overhead in the hot path. That is what turns MLA from a memory trick with a compute tax into a strict serving improvement, and it is why an engineer needs to see the algebra rather than trust the diagram.
The architecture: every piece explained
Top row: the KV compression path. Each token's hidden state h_t enters attention. Instead of projecting straight to per-head keys and values, MLA first applies a down-projection that maps h_t to a small latent vector c_KV of dimension d_c, where d_c is much smaller than num_heads × head_dim. This latent c_KV is the only per-token KV state that is cached. When attention needs the actual keys and values, an up-projection expands c_KV back into per-head keys K and values V — one slice of the up-projection matrix per head, so each head recovers its own distinct K and V from the shared latent. The compression is low-rank: the model learns, during training, to pack the information all heads need into that d_c-dimensional bottleneck.
Middle row: positions and queries. The decoupled RoPE path solves the position problem described above. Alongside the compressed latent, MLA maintains a small, separate key of modest dimension that receives the rotary position embedding directly and is cached as-is; this RoPE key is concatenated with the reconstructed (non-positional) key so that position information rides on a channel that was never compressed. The query path mirrors the key/value path at the query side — queries also go through their own down- and up-projections (a low-rank query compression that reduces activation memory during training) and have their own decoupled RoPE component. Attention itself is then the ordinary per-head softmax(QKᵀ/√d)·V, computed head by head, so MLA is a drop-in replacement for the attention block, not a change to the surrounding transformer.
The absorbed-weights box is where inference efficiency comes from. Because attention scores are (up-projected query) · (up-projected key)ᵀ, and the output is (attention · up-projected value) · output-projection, the up-projection matrices sit in bilinear positions that let them be pre-multiplied into the query projection and the output projection respectively. At inference the engine therefore never materializes the full K and V from the latent — it attends the absorbed queries directly against the cached latents — so the cache is small and the reconstruction cost vanishes from the decoding loop.
Bottom rows: the comparisons and the budget. The KV cache = latent only box is the headline: per token per layer, MLA stores one d_c-dimensional latent plus one small RoPE key, versus MHA's full per-head K and V. The vs MHA/GQA/MQA box frames the distinction — MQA and GQA shrink the cache by dropping or sharing heads (losing expressivity), while MLA shrinks it by compressing information (keeping per-head views). The ops strip names the real levers: the latent rank d_c (the central quality/memory knob), the decoupled RoPE dimension, the resulting cache-size budget per token, and the throughput and concurrency MLA buys you at long context.
End-to-end flow
Trace one decoding step in a served MLA model that has already processed a long prompt. The KV cache holds, for each layer and each prior token, a compact latent c_KV and a small decoupled RoPE key — nothing else. A new token arrives to be generated.
Building the new token's state: the token's hidden state h_t is down-projected to its latent c_KV and to its query latent; the decoupled RoPE keys and query components receive the rotary rotation for this position. The new c_KV and RoPE key are appended to the cache — that append is tiny, a d_c-vector plus a small key, which is exactly why long generations stay affordable in memory.
Computing attention against history: here the weight absorption pays off. Rather than up-projecting every cached latent back into full per-head keys (which would be expensive and would defeat the memory savings by materializing large tensors), the engine uses the query projection with the key up-projection already absorbed into it, so the current query attends directly against the cached latents. The decoupled RoPE key contributes the position-dependent term of the attention score on its separate channel. For each head, the score is the sum of the compressed-content dot product and the RoPE positional dot product; softmax over all history gives the attention weights.
Producing the output: the attention weights combine the cached latents (with the value up-projection absorbed into the output projection) to produce each head's output, which are concatenated and projected to the next hidden state. At no point did the decoding loop materialize full-size keys or values from the latent — the small cache and the absorbed weights kept both memory and compute low. The token is emitted, its latent joins the cache, and the loop repeats.
What this buys at the fleet level: because each token costs a fraction of MHA's cache, a serving node can hold far more concurrent sequences, or far longer contexts per sequence, in the same memory. The binding constraint on throughput — how many KV entries fit — is relaxed by the compression ratio, so effective batch size and context length rise together. And because the per-head reconstruction still gives each head a distinct K and V, the quality does not pay the head-sharing tax that MQA and GQA levy. The stress case to watch is the latent rank: if d_c is set too small, the low-rank bottleneck cannot carry what the heads need and quality degrades; set generously, MLA's cache is still far below MHA's while quality holds — which is exactly the tuning the operational section is about.