Why it matters

MQA was first step toward efficient attention. Understanding shapes evolution to GQA.

Advertisement

The architecture

Q: H heads.

KV: 1 shared.

H Q heads share one KV.

MQA structureQ headsH totalKV heads1 sharedComputeexpand KV to Q shapePaLM used MQA; quality gap led to GQA adoption in Llama, Mistral
MQA.
Advertisement

How it works end to end

KV cache: 1/H of MHA.

Quality: notable degradation.

Replaced by GQA in most modern LLMs.