Why it matters
KV cache is often the memory bottleneck in inference. GQA/MQA lets you serve longer contexts or larger batches for the same GPU. This is why nearly every modern LLM uses one of these.
The architecture
Multi-head: h heads, each with own K, V. Cache = 2 × N × d/h × h × layers = 2 × N × d × layers.
MQA: one K, V shared. Cache = 2 × N × d/h × layers. h times smaller.
GQA: K, V shared across groups of g query heads. Cache size is between MQA and MHA by factor of g.
How it works end to end
Quality: MQA loses more quality than GQA. GQA with g=8 (Llama 2 70B pattern) is nearly indistinguishable from MHA.
Training: models trained with GQA/MQA from scratch work best. Post-hoc conversion (via uptraining) works but with some loss.
Inference savings: proportional to KV reduction. Larger batches, longer context, or smaller GPUs become feasible.