The Geometry of Dot Products

The dot product is the atom of attention

Before any geometry, fix why this matters. Self-attention computes, for a sequence of tokens, a set of scores that say how relevant each token is to each other token. Those scores are literally dot products: each token emits a query vector q and a key vector k, and the relevance of key j to query i is score_ij = q_i · k_j. Stack all queries into a matrix Q: [N, d_k] and all keys into K: [N, d_k] and the whole grid of scores is the matrix product QK^T: [N, N] — one dot product per entry.

Everything else in attention is bookkeeping around this core: a softmax to turn scores into weights, a weighted sum of value vectors to produce the output. The intelligence — the ‘which words belong together’ — lives in the dot product. So the question ‘why does attention work?’ is really the question ‘why does a dot product measure similarity?’ And to answer that we need to see the dot product not as a formula but as a piece of geometry. That is the whole journey of this piece: make q·k mean something you can picture.

Two definitions that turn out to be the same

The dot product has two faces. The algebraic definition is the one you compute with: for vectors a and b with d coordinates each,

a · b = a_1·b_1 + a_2·b_2 + ... + a_d·b_d = Σ_i a_i b_i

Multiply matching coordinates, add them up, get a single number (a scalar). For a = (3, 4) and b = (4, 3): a·b = 3×4 + 4×3 = 12 + 12 = 24.

The geometric definition is the one you reason with:

a · b = |a| · |b| · cosθ

where |a| and |b| are the lengths (norms) of the vectors and θ is the angle between them. This says the dot product is big when the vectors are long and point the same way, zero when they are perpendicular, and negative when they point in opposing directions. That these two definitions — a coordinate sum on one side, lengths and an angle on the other — always agree is the small miracle that makes the dot product useful. The next section proves why.

Why the geometric identity holds

The bridge between the two definitions is the law of cosines, the generalization of Pythagoras to non-right triangles. Consider the triangle formed by vectors a, b, and the side connecting their tips, a − b. The law of cosines relates the three side lengths and the angle θ between a and b:

|a − b|^2 = |a|^2 + |b|^2 − 2·|a||b|·cosθ

Now expand the left side algebraically, using the fact that |v|^2 = v·v:

|a − b|^2 = (a−b)·(a−b)
            = a·a − 2(a·b) + b·b
            = |a|^2 − 2(a·b) + |b|^2

Set the two expressions for |a−b|^2 equal. The |a|^2 and |b|^2 terms cancel from both sides, leaving −2(a·b) = −2|a||b|cosθ, i.e. a·b = |a||b|cosθ. The algebraic coordinate sum and the geometric length-times-cosine are provably the same quantity. This is not a definition we adopt by convention; it is forced by the geometry of triangles, which is exactly why the dot product carries real geometric information.

The dot product as a projection (a shadow)

The cleanest mental picture of a dot product is a shadow. Rearrange the identity: a·b = |b| × (|a|cosθ). The quantity |a|cosθ is the length of a’s shadow when you shine a light perpendicular onto the line through b — the scalar projection of a onto b. So the dot product measures ‘how much of a points along b,’ scaled by the length of b.

Worked example: keep a = (3, 4), b = (4, 3). Both have length 5 (since √(9+16) = 5). We found a·b = 24. The scalar projection of a onto b is a·b / |b| = 24 / 5 = 4.8 — nearly the full length 5 of a, because the two vectors point in almost the same direction. If instead b pointed at right angles to a, the shadow would collapse to zero and so would the dot product. This projection view is why attention can be read as ‘how much does this key lie along the direction the query is asking about?’ A query is a question posed as a direction; the dot product reports how far each key extends in that direction.

The sign and size of the score, decoded

Because |a| and |b| are always non-negative, the sign of a dot product is entirely the sign of cosθ, and that gives a clean three-way reading of any attention score:

So a raw attention score blends two things: the angle (do these vectors point the same way?) and the magnitudes (how long are they?). A large positive score can come from near-perfect alignment of modest vectors, or from mediocre alignment of very long vectors. This entanglement of angle and length is important and slightly dangerous: it is the reason the raw dot product and cosine similarity are not the same measurement, and the reason a model can use vector length as a lever to make some tokens systematically louder than others regardless of their direction. We pull those two apart next.

Vector norms: the length half of the story

The norm (or magnitude, or length) of a vector is |v| = √(v·v) = √(Σ_i v_i^2) — the straight-line distance from the origin to the vector’s tip, the d-dimensional Pythagoras. Norms carry real signal in a transformer. Because the dot product scales linearly with each norm, doubling the length of a key doubles every score that key participates in. A token whose key vector is long will tend to win attention from everyone, purely on magnitude, before direction is even considered.

Numeric feel: take u = (1, 0) and two candidates, v = (1, 1) and w = (3, 3). Directionally v and w are identical (both at 45° to u), yet u·v = 1 while u·w = 3. The extra score is pure length: |w| = 3√2 is three times |v| = √2. This is why practitioners watch embedding and key norms, why layer normalization sits where it does, and why the distinction between measuring direction and measuring direction-and-length is not pedantic. Sometimes you want length to matter (a token really should be more prominent); sometimes it is noise you want to divide out. That choice is exactly cosine similarity versus the raw dot product.

Cosine similarity versus the raw dot product

Cosine similarity is the dot product with the magnitudes divided out — it keeps only the angle:

cosθ = (a · b) / (|a| · |b|)     →  always in [−1, +1]

It answers ‘which direction?’ and ignores ‘how long?’ The raw dot product answers both at once. When are they different? Whenever the norms differ. Consider a query q = (1, 1) comparing two keys:

k1 = (1, 1):   q·k1 = 2     cos = 2/(√2·√2)   = 1.00
k2 = (5, 5):   q·k2 = 10    cos = 10/(√2·√50)  = 1.00

By cosine, k1 and k2 are equally relevant — same direction, similarity 1.0. By raw dot product, k2 scores five times higher, entirely because it is five times longer. Neither answer is ‘wrong’; they measure different things. Attention deliberately uses the raw dot product, so magnitude is a usable signal — the model can learn to make certain keys longer to boost their reach. Retrieval and embedding-search systems usually prefer cosine (or normalize vectors first) because there you want pure semantic direction, not an artifact of how confidently some vector happened to be scaled. Knowing which you are computing is half of debugging a similarity system.

QK^T: relevance as a grid of dot products

Now assemble the attention scores. Each token’s embedding x is projected into a query and a key by learned matrices: q = x W_Q, k = x W_K, each of dimension d_k. Stacking a sequence of N tokens gives Q: [N, d_k] and K: [N, d_k], and the full score matrix is S = QK^T: [N, N], where S_ij = q_i · k_j.

Read geometrically: W_Q and W_K rotate and stretch the shared embedding space into a space where the ‘question’ directions and the ‘answer’ directions line up when tokens are relevant. Training tunes those projections so that, for pairs that should attend, the query and key end up pointing similar ways (large positive dot product), and for irrelevant pairs they end up near-perpendicular (dot product near zero). The softmax over each row then turns a row of raw scores into a probability distribution — the attention weights — and the output is the weighted sum of value vectors. Crucially, the only place tokens actually compare is that dot product. Every bit of ‘this word relates to that word’ is a geometric alignment the projections learned to create. The matrix QK^T is just N×N shadows, computed all at once.

A fully worked single-head example

Make it concrete with d_k = 4. Suppose one query and three keys come out of the projections as:

q  = ( 1,  2,  0,  1)
k1 = ( 1,  2,  1,  0)   (a related token)
k2 = (−1, −2,  0,  1)   (an opposed token)
k3 = ( 0,  0,  1,  0)   (an unrelated token)

q·k1 = 1 + 4 + 0 + 0 =  5
q·k2 = −1 − 4 + 0 + 1 = −4
q·k3 = 0 + 0 + 0 + 0 =  0

The scores (5, −4, 0) already tell the story: k1 aligns with the query, k2 opposes it, k3 is perpendicular (irrelevant). Push them through softmax (here without scaling, for clarity): exp(5)=148.4, exp(−4)=0.018, exp(0)=1, summing to 149.4. The attention weights are (0.993, 0.0001, 0.0067). The query pours essentially all of its attention onto k1, almost none onto the opposed or unrelated tokens. That is the entire mechanism: dot products rank the keys, softmax sharpens the ranking into weights, and the output becomes a near-copy of k1’s value vector. No magic — just projections and shadows.

Why dot products grow with √d

Here is a subtlety that becomes a real engineering problem. Suppose the coordinates of q and k are independent random numbers with mean 0 and variance 1 (a decent model of freshly projected vectors). What is the typical size of q·k = Σ_i q_i k_i?

Each term q_i k_i has mean 0 (the coordinates are independent, so E[q_i k_i] = E[q_i]E[k_i] = 0) and variance Var(q_i k_i) = E[q_i^2]E[k_i^2] = 1×1 = 1. Summing d_k independent terms, the variances add: Var(q·k) = d_k. So the standard deviation — the typical magnitude — of the dot product is √d_k, not d_k and not a constant. The score does not stay a nice O(1) number as you widen the head; it drifts outward like the square root of the dimension. For d_k = 64 the scores have spread √64 = 8; for d_k = 512, spread √512 ≈ 22.6. Bigger heads produce systematically bigger raw scores, purely from adding up more random terms — and that has an unpleasant consequence for softmax, which the next section fixes.

The 1/√d_k scaling, motivated

Why does that √d_k drift matter? Because softmax is exponential, and exponentials are extremely sensitive to the scale of their inputs. If the scores span a range like ±20 instead of ±3, the softmax becomes nearly one-hot: the single largest score gets a weight of essentially 1 and everything else gets ≈0. A saturated softmax has vanishing gradients almost everywhere — the model can barely learn, because tiny changes to the scores no longer move the weights. Large d_k would silently break training.

The fix is exactly to cancel the drift we derived. Divide every score by √d_k before the softmax:

Attention(Q,K,V) = softmax( QK^T / √d_k ) V

Since the raw scores have standard deviation √d_k, dividing by √d_k restores their spread to about 1 regardless of head width. The softmax then operates in a sane range at any dimension, gradients stay healthy, and a 64-wide head and a 512-wide head behave comparably. This is the origin of the famous 1/√d_k in ‘Attention Is All You Need’ — not an arbitrary constant but the precise factor that undoes the √d growth of a d-term dot product.

Orthogonality: the geometry of , '’': unrelated’

The identity a·b = |a||b|cosθ gives a crisp definition of ‘unrelated’: two non-zero vectors are orthogonal (perpendicular) exactly when their dot product is 0, because that forces cosθ = 0 and θ = 90°. Orthogonal directions are independent axes of meaning: activity along one contributes nothing to a score measured along the other. In attention terms, a key that is orthogonal to a query contributes a score of exactly zero — it is neither supported nor opposed, simply off-topic.

This is why models like to encode distinct concepts along orthogonal (or nearly orthogonal) directions: features stored on perpendicular axes do not interfere, so a query asking about one concept does not accidentally light up another. The classic example is a standard basis: (1,0,0), (0,1,0), (0,0,1) are mutually orthogonal, and in d dimensions you can find exactly d mutually perpendicular directions. If a model could only use strictly orthogonal directions, a 512-dimensional space could hold at most 512 clean, non-interfering concepts. Real models store far more than that — and the reason they can is the beautiful statistics of high dimensions in the next section.