Why architecture matters here
The architecture matters because conditioning is the whole game in sequence-to-sequence and multimodal tasks, and cross-attention is the cleanest way a transformer expresses conditioning. When you translate, summarize, caption, or answer from retrieved documents, the output is not a free-standing generation — it must be grounded in a specific input. Cross-attention makes that grounding differentiable and learnable: the model learns, per output position, which parts of the source to weight, rather than being forced to compress the entire source into a single fixed vector as pre-attention encoder-decoders did.
It matters because it solved the information bottleneck that crippled early neural machine translation. The original sequence-to-sequence models squeezed the whole source sentence into one final hidden state and asked the decoder to generate from that alone; long sentences lost information at the far end because everything had to survive a single vector. Cross-attention removed the bottleneck by letting the decoder attend over all the encoder's per-token states directly, so no source position is ever more than one attention hop away from any output position. That is the change that made attention-based translation leap past its predecessors.
It matters because of an efficiency property that shapes how you build with it: the source is encoded once, and its keys and values are reused across every target step. In an encoder-decoder, the encoder runs a single forward pass over the source; the resulting K and V are fixed for the entire decode. Every generated token cross-attends to the same cached source K/V, so the marginal cost of conditioning on a long source is paid once, not per output token. This asymmetry — cheap to reuse a fixed context — is exactly why cross-attention fits retrieval and multimodal fusion, where the external context is large but static during generation.
It matters because it draws a clean architectural boundary between the model's two jobs. In an encoder-decoder, self-attention within the decoder handles fluency and coherence of the output so far, while cross-attention handles faithfulness to the source. Separating those concerns into distinct attention layers makes the model easier to reason about and to diagnose: a translation that is fluent but wrong is a cross-attention (grounding) failure, while one that is faithful but garbled is a self-attention (generation) failure. That interpretability is a real engineering asset.
Finally it matters because knowing when not to use it is as important as knowing when to. Decoder-only models deliberately fold the context into a single self-attention stream by concatenating source and target into one sequence, trading cross-attention's efficiency for architectural uniformity and the ability to share a KV cache across the whole prompt. Understanding cross-attention's costs and benefits is what lets you choose between an encoder-decoder and a decoder-only design for a given task instead of cargo-culting one or the other.
The architecture: every piece explained
Begin with the two sequences. The target is the stream the model is producing or refining — the decoder's tokens generated so far, or a set of query tokens. The source is the external context to condition on — the encoder's output over an input sentence, the patch embeddings of an image, the encoded representations of retrieved passages. These are genuinely different tensors, often of different lengths and sometimes of different modalities, and keeping straight which one supplies which projection is the key to the whole operation.
The query projection comes from the target. Each target position is multiplied by a learned weight matrix to produce a query vector that encodes 'what information am I, this output position, looking for?' Because queries come from the target, there is exactly one query per target position, and the number of queries equals the target length — which is why cross-attention output has the target's shape, not the source's.
The key and value projections come from the source. The source sequence is multiplied by two more learned matrices to produce a key and a value per source position. Keys encode 'what information do I, this source position, offer?' and values encode the actual content to be pulled forward. Because K and V come from the source, there are as many keys and values as there are source positions, and crucially they do not change as the target grows — the source is fixed, so K and V can be computed once and cached.
The scaled dot-product is the matching step, unchanged from self-attention. Each target query is dotted with every source key, scaled by the square root of the head dimension to keep the softmax well-conditioned, and passed through a softmax to produce a distribution over source positions. This distribution — the attention weights — says, for this one output position, how much to draw from each source position. A translation model, generating the word 'house,' will typically put most of its weight on the source position holding 'maison.'
The weights then form the context vector as a weighted sum of the source values, and that context is fused back into the target stream through the usual residual add and layer normalization, followed by the block's feed-forward network. In a multi-head setup this all happens in parallel across several heads, each free to specialize — one head tracking positional alignment, another semantic correspondence — and their outputs are concatenated and projected. The block's output has the target's length but is now infused with source information: the target has read from the context.
End-to-end flow
Walk a full encoder-decoder translation. First the encoder runs its self-attention stack over the entire source sentence once, producing a sequence of contextual hidden states — one per source token. From those, the decoder's cross-attention layers precompute the keys and values, which are now fixed for the whole generation. This is the one-time cost of ingesting the source.
Now decoding begins, autoregressively. The decoder holds the tokens generated so far (initially just a start token). In each decoder block, a self-attention layer runs first, letting the output-so-far attend over itself (with a causal mask so no position sees the future) to maintain fluency. Then the cross-attention layer runs: it projects the current decoder states into queries and dots them against the cached source keys, producing, for each output position, a distribution over the source tokens.
Suppose the decoder is about to generate the third word of the translation. Its query for that position lights up the source keys most relevant to what comes next — perhaps a noun near the middle of the source sentence — and the softmax concentrates weight there. The context vector is a blend dominated by that source token's value, and it flows through the residual and feed-forward layers to bias the output distribution toward the correct target word. The model has, in effect, looked at the source and pulled the relevant piece forward.
The step completes, a token is sampled, and it is appended to the decoder's sequence. The next step repeats: self-attention over the now-longer output, cross-attention against the same cached source K/V, another context vector, another token. The source is never re-encoded; only the target grows. Over the full generation, the cross-attention weights trace an implicit alignment between output and source positions — the learned analogue of a word-alignment matrix — which is why attention heatmaps from these models are so interpretable.
Contrast the multimodal and retrieval cases, which reuse the identical machinery with a different source. In image captioning, the source K/V come from a vision encoder's patch embeddings, so each generated word cross-attends over image regions. In retrieval-augmented generation, the source is the encoded set of retrieved passages, so the generator conditions on external documents. In every case the pattern is the same: encode the context once into keys and values, then let the target stream cross-attend to it repeatedly as it generates, paying the context cost a single time and reading from it as often as needed.