Why it matters
The seemingly-arbitrary sqrt(d_k) factor is essential. Without it, transformers with large hidden dimensions would train poorly. Understanding why deepens intuition about softmax behavior.
The architecture
Formula: Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V.
Where d_k is the dimensionality of the key vectors.
How it works end to end
Why scale: if Q and K components have variance 1, dot product has variance d_k. For large d_k (128, 256), dot products are large. Softmax on large inputs saturates — one weight goes to 1, others to 0. Gradients vanish.
Scaling by sqrt(d_k) keeps dot product variance around 1. Softmax stays in a well-behaved regime.
Dot product vs additive attention: dot product is much more parallelizable on GPU (single matmul). Additive attention (Bahdanau) uses feedforward network — slower.