Why it matters
Multi-head is what gives transformers their representational richness. Single-head attention is limited; multi-head lets the model learn many attention patterns in parallel.
The architecture
Given d-dimensional input, project to h heads of d/h dimensions each. Each head runs scaled dot-product attention independently. Concatenate outputs (h × d/h = d dimensions). Final linear projection.
Typical: d=512, h=8, so each head is 64-dim.
How it works end to end
Parameters per head: W_Q, W_K, W_V matrices of size (d, d/h) each. Total parameters same as single head with d dimensions.
Interpretability: analysis of trained models shows heads specialize — some attend to next word, some to syntactic children, some to semantic similarity.
Grouped-query attention (GQA): share K, V across groups of heads to reduce memory. Compromise between full multi-head and multi-query.