The Geometry of Dot Products

The dot product is the atom of attention

Before any geometry, fix why this matters. Self-attention computes, for a sequence of tokens, a set of scores that say how relevant each token is to each other token. Those scores are literally dot products: each token emits a query vector q and a key vector k, and the relevance of key j to query i is score_ij = q_i · k_j. Stack all queries into a matrix Q: [N, d_k] and all keys into K: [N, d_k] and the whole grid of scores is the matrix product QK^T: [N, N] — one dot product per entry.

Everything else in attention is bookkeeping around this core: a softmax to turn scores into weights, a weighted sum of value vectors to produce the output. The intelligence — the ‘which words belong together’ — lives in the dot product. So the question ‘why does attention work?’ is really the question ‘why does a dot product measure similarity?’ And to answer that we need to see the dot product not as a formula but as a piece of geometry. That is the whole journey of this piece: make q·k mean something you can picture.

Two definitions that turn out to be the same

The dot product has two faces. The algebraic definition is the one you compute with: for vectors a and b with d coordinates each,

a · b = a_1·b_1 + a_2·b_2 + ... + a_d·b_d = Σ_i a_i b_i

Multiply matching coordinates, add them up, get a single number (a scalar). For a = (3, 4) and b = (4, 3): a·b = 3×4 + 4×3 = 12 + 12 = 24.

The geometric definition is the one you reason with:

a · b = |a| · |b| · cosθ

where |a| and |b| are the lengths (norms) of the vectors and θ is the angle between them. This says the dot product is big when the vectors are long and point the same way, zero when they are perpendicular, and negative when they point in opposing directions. That these two definitions — a coordinate sum on one side, lengths and an angle on the other — always agree is the small miracle that makes the dot product useful. The next section proves why.

Why the geometric identity holds

The bridge between the two definitions is the law of cosines, the generalization of Pythagoras to non-right triangles. Consider the triangle formed by vectors a, b, and the side connecting their tips, a − b. The law of cosines relates the three side lengths and the angle θ between a and b:

|a − b|^2 = |a|^2 + |b|^2 − 2·|a||b|·cosθ

Now expand the left side algebraically, using the fact that |v|^2 = v·v:

|a − b|^2 = (a−b)·(a−b)
            = a·a − 2(a·b) + b·b
            = |a|^2 − 2(a·b) + |b|^2

Set the two expressions for |a−b|^2 equal. The |a|^2 and |b|^2 terms cancel from both sides, leaving −2(a·b) = −2|a||b|cosθ, i.e. a·b = |a||b|cosθ. The algebraic coordinate sum and the geometric length-times-cosine are provably the same quantity. This is not a definition we adopt by convention; it is forced by the geometry of triangles, which is exactly why the dot product carries real geometric information.

The dot product as a projection (a shadow)

The cleanest mental picture of a dot product is a shadow. Rearrange the identity: a·b = |b| × (|a|cosθ). The quantity |a|cosθ is the length of a’s shadow when you shine a light perpendicular onto the line through b — the scalar projection of a onto b. So the dot product measures ‘how much of a points along b,’ scaled by the length of b.

Worked example: keep a = (3, 4), b = (4, 3). Both have length 5 (since √(9+16) = 5). We found a·b = 24. The scalar projection of a onto b is a·b / |b| = 24 / 5 = 4.8 — nearly the full length 5 of a, because the two vectors point in almost the same direction. If instead b pointed at right angles to a, the shadow would collapse to zero and so would the dot product. This projection view is why attention can be read as ‘how much does this key lie along the direction the query is asking about?’ A query is a question posed as a direction; the dot product reports how far each key extends in that direction.

The sign and size of the score, decoded

Because |a| and |b| are always non-negative, the sign of a dot product is entirely the sign of cosθ, and that gives a clean three-way reading of any attention score:

So a raw attention score blends two things: the angle (do these vectors point the same way?) and the magnitudes (how long are they?). A large positive score can come from near-perfect alignment of modest vectors, or from mediocre alignment of very long vectors. This entanglement of angle and length is important and slightly dangerous: it is the reason the raw dot product and cosine similarity are not the same measurement, and the reason a model can use vector length as a lever to make some tokens systematically louder than others regardless of their direction. We pull those two apart next.

Vector norms: the length half of the story

The norm (or magnitude, or length) of a vector is |v| = √(v·v) = √(Σ_i v_i^2) — the straight-line distance from the origin to the vector’s tip, the d-dimensional Pythagoras. Norms carry real signal in a transformer. Because the dot product scales linearly with each norm, doubling the length of a key doubles every score that key participates in. A token whose key vector is long will tend to win attention from everyone, purely on magnitude, before direction is even considered.

Numeric feel: take u = (1, 0) and two candidates, v = (1, 1) and w = (3, 3). Directionally v and w are identical (both at 45° to u), yet u·v = 1 while u·w = 3. The extra score is pure length: |w| = 3√2 is three times |v| = √2. This is why practitioners watch embedding and key norms, why layer normalization sits where it does, and why the distinction between measuring direction and measuring direction-and-length is not pedantic. Sometimes you want length to matter (a token really should be more prominent); sometimes it is noise you want to divide out. That choice is exactly cosine similarity versus the raw dot product.

Cosine similarity versus the raw dot product

Cosine similarity is the dot product with the magnitudes divided out — it keeps only the angle:

cosθ = (a · b) / (|a| · |b|)     →  always in [−1, +1]

It answers ‘which direction?’ and ignores ‘how long?’ The raw dot product answers both at once. When are they different? Whenever the norms differ. Consider a query q = (1, 1) comparing two keys:

k1 = (1, 1):   q·k1 = 2     cos = 2/(√2·√2)   = 1.00
k2 = (5, 5):   q·k2 = 10    cos = 10/(√2·√50)  = 1.00

By cosine, k1 and k2 are equally relevant — same direction, similarity 1.0. By raw dot product, k2 scores five times higher, entirely because it is five times longer. Neither answer is ‘wrong’; they measure different things. Attention deliberately uses the raw dot product, so magnitude is a usable signal — the model can learn to make certain keys longer to boost their reach. Retrieval and embedding-search systems usually prefer cosine (or normalize vectors first) because there you want pure semantic direction, not an artifact of how confidently some vector happened to be scaled. Knowing which you are computing is half of debugging a similarity system.

QK^T: relevance as a grid of dot products

Now assemble the attention scores. Each token’s embedding x is projected into a query and a key by learned matrices: q = x W_Q, k = x W_K, each of dimension d_k. Stacking a sequence of N tokens gives Q: [N, d_k] and K: [N, d_k], and the full score matrix is S = QK^T: [N, N], where S_ij = q_i · k_j.

Read geometrically: W_Q and W_K rotate and stretch the shared embedding space into a space where the ‘question’ directions and the ‘answer’ directions line up when tokens are relevant. Training tunes those projections so that, for pairs that should attend, the query and key end up pointing similar ways (large positive dot product), and for irrelevant pairs they end up near-perpendicular (dot product near zero). The softmax over each row then turns a row of raw scores into a probability distribution — the attention weights — and the output is the weighted sum of value vectors. Crucially, the only place tokens actually compare is that dot product. Every bit of ‘this word relates to that word’ is a geometric alignment the projections learned to create. The matrix QK^T is just N×N shadows, computed all at once.

A fully worked single-head example

Make it concrete with d_k = 4. Suppose one query and three keys come out of the projections as:

q  = ( 1,  2,  0,  1)
k1 = ( 1,  2,  1,  0)   (a related token)
k2 = (−1, −2,  0,  1)   (an opposed token)
k3 = ( 0,  0,  1,  0)   (an unrelated token)

q·k1 = 1 + 4 + 0 + 0 =  5
q·k2 = −1 − 4 + 0 + 1 = −4
q·k3 = 0 + 0 + 0 + 0 =  0

The scores (5, −4, 0) already tell the story: k1 aligns with the query, k2 opposes it, k3 is perpendicular (irrelevant). Push them through softmax (here without scaling, for clarity): exp(5)=148.4, exp(−4)=0.018, exp(0)=1, summing to 149.4. The attention weights are (0.993, 0.0001, 0.0067). The query pours essentially all of its attention onto k1, almost none onto the opposed or unrelated tokens. That is the entire mechanism: dot products rank the keys, softmax sharpens the ranking into weights, and the output becomes a near-copy of k1’s value vector. No magic — just projections and shadows.

Why dot products grow with √d

Here is a subtlety that becomes a real engineering problem. Suppose the coordinates of q and k are independent random numbers with mean 0 and variance 1 (a decent model of freshly projected vectors). What is the typical size of q·k = Σ_i q_i k_i?

Each term q_i k_i has mean 0 (the coordinates are independent, so E[q_i k_i] = E[q_i]E[k_i] = 0) and variance Var(q_i k_i) = E[q_i^2]E[k_i^2] = 1×1 = 1. Summing d_k independent terms, the variances add: Var(q·k) = d_k. So the standard deviation — the typical magnitude — of the dot product is √d_k, not d_k and not a constant. The score does not stay a nice O(1) number as you widen the head; it drifts outward like the square root of the dimension. For d_k = 64 the scores have spread √64 = 8; for d_k = 512, spread √512 ≈ 22.6. Bigger heads produce systematically bigger raw scores, purely from adding up more random terms — and that has an unpleasant consequence for softmax, which the next section fixes.

The 1/√d_k scaling, motivated

Why does that √d_k drift matter? Because softmax is exponential, and exponentials are extremely sensitive to the scale of their inputs. If the scores span a range like ±20 instead of ±3, the softmax becomes nearly one-hot: the single largest score gets a weight of essentially 1 and everything else gets ≈0. A saturated softmax has vanishing gradients almost everywhere — the model can barely learn, because tiny changes to the scores no longer move the weights. Large d_k would silently break training.

The fix is exactly to cancel the drift we derived. Divide every score by √d_k before the softmax:

Attention(Q,K,V) = softmax( QK^T / √d_k ) V

Since the raw scores have standard deviation √d_k, dividing by √d_k restores their spread to about 1 regardless of head width. The softmax then operates in a sane range at any dimension, gradients stay healthy, and a 64-wide head and a 512-wide head behave comparably. This is the origin of the famous 1/√d_k in ‘Attention Is All You Need’ — not an arbitrary constant but the precise factor that undoes the √d growth of a d-term dot product.

Orthogonality: the geometry of , '’': unrelated’

The identity a·b = |a||b|cosθ gives a crisp definition of ‘unrelated’: two non-zero vectors are orthogonal (perpendicular) exactly when their dot product is 0, because that forces cosθ = 0 and θ = 90°. Orthogonal directions are independent axes of meaning: activity along one contributes nothing to a score measured along the other. In attention terms, a key that is orthogonal to a query contributes a score of exactly zero — it is neither supported nor opposed, simply off-topic.

This is why models like to encode distinct concepts along orthogonal (or nearly orthogonal) directions: features stored on perpendicular axes do not interfere, so a query asking about one concept does not accidentally light up another. The classic example is a standard basis: (1,0,0), (0,1,0), (0,0,1) are mutually orthogonal, and in d dimensions you can find exactly d mutually perpendicular directions. If a model could only use strictly orthogonal directions, a 512-dimensional space could hold at most 512 clean, non-interfering concepts. Real models store far more than that — and the reason they can is the beautiful statistics of high dimensions in the next section.

Near-orthogonality: high dimensions are mostly perpendicular

Here is the fact that makes large embedding spaces so powerful, and it is genuinely counter-intuitive: in high dimensions, two random vectors are almost always nearly perpendicular. Take two vectors with random unit-scale coordinates. Their dot product has mean 0 and, from the argument above, standard deviation √d relative to norms of about √d each — so the cosine of the angle between them has typical size √d / (√d · √d) = 1/√d, which shrinks toward zero as d grows.

Concretely, in d = 1000 dimensions the typical cosine between two random vectors is about 1/√1000 ≈ 0.032, an angle of roughly 88–89° — essentially a right angle. Random vectors do not point ‘somewhat’ apart; they point almost exactly perpendicular. The consequence is enormous: while only d vectors can be exactly orthogonal, the number of vectors that are approximately orthogonal (within a few degrees) grows exponentially with d. A 512- or 4096-dimensional space can therefore host far, far more than d nearly-independent concept directions, each of which barely interferes with the others under a dot product. This is the geometric permission slip for ‘superposition’ — how a model crams thousands of features into a few thousand dimensions.

What this buys transformers (and CPU SLMs)

Fold the pieces together and the design of attention stops looking arbitrary. The dot product is chosen because it is the cheap, differentiable operation that measures directional alignment; the 1/√d_k keeps that measurement well-scaled at any head width; and near-orthogonality in high dimensions is what lets a modest embedding size carry a huge vocabulary of meanings without them colliding. All three are consequences of the same geometry.

For small language models running on a CPU, the practical lesson is that dot-product attention is astonishingly cheap per comparison — d_k multiply-adds, no branches, perfectly vectorizable — which is exactly why it maps well onto SIMD units and cache-friendly matrix kernels. The cost is not the dot product itself but the N×N grid of them (the quadratic-in-sequence-length problem covered elsewhere in this series). It also means the levers you have are geometric: normalizing embeddings changes whether magnitude or direction dominates; choosing d_k trades off representational room against compute and against the √d_k scaling; and keeping key norms controlled prevents a few tokens from hijacking attention on magnitude alone. Think in vectors and angles, and the knobs become obvious.

Common pitfalls and misreadings

A few traps trip people up once they start reasoning with dot products. First, confusing raw dot products with cosine similarity. If you compare vectors of different norms, a bigger raw score can mean ‘longer vector,’ not ‘more similar direction.’ Decide which you want, and normalize if you mean cosine.

Second, forgetting the scaling. Computing QK^T without the 1/√d_k factor at large d_k saturates the softmax and stalls learning — a subtle bug that looks like ‘the model won’t train’ rather than an obvious crash. Third, reading a zero score as ‘dissimilar’. Zero means orthogonal — unrelated, sitting between positive (related) and negative (opposed); a strongly dissimilar, opposed key gives a large negative score, not zero. Fourth, importing low-dimensional intuition. In 2D or 3D, random vectors are often far from perpendicular; in 500D they are almost never far from it. Your mental picture of three axes actively misleads you about what a 512-dimensional embedding space is like. Hold onto the algebra — mean 0, spread √d, cosine ≈ 1/√d — when the geometry gets too big to visualize.