Why attention is order-blind
Start with the property that makes positional encoding necessary. Self-attention computes, for each token, a weighted average of value vectors, where the weights come from query–key dot products: Attention(X) = softmax(QK^T / √d_k) V. Nowhere in that expression does the index of a token appear. Q, K, and V are produced by the same linear maps applied to every position identically, and the softmax sums over all positions symmetrically.
The precise statement is that self-attention is permutation-equivariant, not permutation-invariant. If P is a permutation matrix that reorders the rows of the input, then Attention(PX) = P·Attention(X) — permute the inputs and the outputs permute in lockstep, but no new information about order enters. The practical consequence is what people loosely call ‘order-blindness’: a given token’s output depends only on which tokens are present (the multiset) and not on their arrangement. So the raw attention block genuinely cannot tell ‘dog bites man’ from ‘man bites dog.’ Recurrent and convolutional networks get order for free from their structure; a transformer has to be told where each token is.
The fix: add position into the embedding
The transformer’s answer is disarmingly simple: manufacture a vector that encodes position and add it to the token embedding before the first attention layer. If E is the token embedding for a word and PE(pos) is the positional vector for its slot, the layer input is x = E + PE(pos). Both live in the same d-dimensional space, so the sum is well defined and the model can learn to read either component.
Why add rather than concatenate? Concatenation would grow the dimension and spend parameters keeping the two signals in separate subspaces. Addition keeps d fixed and lets the network allocate directions in the embedding space to position as it sees fit — in practice the learned W_Q and W_K projections can pull the positional part into whichever subspace is useful for computing attention scores. The key requirement is that PE(pos) be a distinct, structured function of pos: distinct so positions are distinguishable, and structured so that nearby positions get similar vectors and the notion of distance is recoverable. Sinusoids satisfy both.
The sinusoidal formula
The original ‘Attention Is All You Need’ encoding defines each coordinate of the positional vector with a sine or cosine whose frequency depends on the dimension. Index the d coordinates in pairs by i = 0, 1, …, d/2 − 1. Then:
PE(pos, 2i) = sin( pos / 10000^(2i/d) )
PE(pos, 2i+1) = cos( pos / 10000^(2i/d) )So even coordinates get sines, odd coordinates get cosines, and the sine–cosine pair at index i shares a single angular frequency ω_i = 1 / 10000^(2i/d). Equivalently, the argument is pos · ω_i, an angle that advances linearly with position at a rate set by the dimension. At i = 0 the denominator is 10000^0 = 1, giving the highest frequency ω = 1 (a fast oscillation, wavelength 2π). As i grows the denominator grows toward 10000, so the frequency shrinks and the wavelength stretches toward 2π · 10000. Each position therefore maps to a unique vector of coordinates — a bank of ‘clocks’ ticking at many different rates, from very fast to very slow.
The 10000^(2i/d) wavelength schedule
The 10000^(2i/d) term is a geometric progression of wavelengths, and that choice is deliberate. Writing the wavelength as λ_i = 2π · 10000^(2i/d), the wavelengths sweep from 2π (fastest, at i = 0) up to about 2π · 10000 (slowest, at i = d/2 − 1), spaced evenly on a log scale rather than linearly.
This is the same trick as a positional number system or the hands of a clock. The fast, high-frequency dimensions distinguish adjacent positions sharply — they change a lot from pos to pos+1. The slow, low-frequency dimensions barely move between neighbors but disambiguate positions that are far apart, giving the encoding range across long sequences without any two positions colliding. Because the frequencies are geometrically spaced, the encoding carries information at every scale simultaneously — a few tokens apart, a few hundred apart, a few thousand apart. The constant 10000 is a hyperparameter setting the slowest wavelength; it was chosen so the longest period comfortably exceeds realistic sequence lengths, ensuring the slowest clock does not wrap around and alias two distant positions to the same phase.
Shapes: how the encoding is applied
Keeping the tensor shapes straight makes the mechanics concrete. Let N be the sequence length and d (often written d_model) the embedding dimension. The token embeddings form a matrix X: [N, d] (or [B, N, d] with a batch axis B). The positional encoding is a matrix PE: [N, d] whose row pos is the vector PE(pos) from the formula.
Applying it is a single elementwise add, broadcast over the batch: X’ = X + PE, still [N, d]. That is the entire operation — no parameters, no matrix multiply. For the fixed sinusoidal scheme the whole PE matrix can be precomputed once and cached, or even computed on the fly, because it depends only on pos and i, never on the data. Since attention is permutation-equivariant, the add must happen before the first attention layer; do it after, and each layer would still be reordering a bag of position-tagged vectors correctly, but the standard design injects position once at the input and lets it propagate. On the CPU-SLM side this is a negligible cost — an [N, d] add against the O(N^2 d) of attention itself — which is one reason fixed encodings remain attractive for small, tight inference loops.
A worked numeric example
Take a tiny model with d = 4, so i ∈ {0, 1} and there are two frequency pairs. The frequencies are ω_0 = 1 / 10000^(0/4) = 1 and ω_1 = 1 / 10000^(2/4) = 1/100 = 0.01. Coordinates 0 and 1 use ω_0; coordinates 2 and 3 use ω_1.
PE(pos) = [ sin(pos·1), cos(pos·1), sin(pos·0.01), cos(pos·0.01) ]
pos=0: [ sin(0), cos(0), sin(0), cos(0) ] = [ 0.000, 1.000, 0.000, 1.000 ]
pos=1: [ sin(1), cos(1), sin(0.01), cos(0.01) ] = [ 0.841, 0.540, 0.010, 1.000 ]
pos=2: [ sin(2), cos(2), sin(0.02), cos(0.02) ] = [ 0.909, -0.416, 0.020, 1.000 ]Notice the structure. The first pair (the fast clock) swings dramatically between positions — from (0, 1) to (0.841, 0.540) to (0.909, −0.416) — so it sharply separates neighbors. The second pair (the slow clock) barely moves: 0.010 then 0.020, its cosine still essentially 1. Over these three positions the slow clock looks almost constant, but across hundreds of positions it is what keeps distant tokens distinct. Every row is a unique 4-vector, so no two positions collide.
Why sinusoids: relative position is a linear rotation
Here is the property that justifies the whole design. Fix a frequency ω and look at its sine–cosine pair as a 2D vector. Shifting the position by a constant offset k is exactly a rotation of that vector by the angle ωk — and crucially the rotation does not depend on pos. From the angle-addition identities:
sin(ω(pos+k)) = sin(ω·pos)·cos(ωk) + cos(ω·pos)·sin(ωk)
cos(ω(pos+k)) = cos(ω·pos)·cos(ωk) − sin(ω·pos)·sin(ωk)In matrix form, [sin(ω(pos+k)), cos(ω(pos+k))]^T = R(ωk) · [sin(ω·pos), cos(ω·pos)]^T, where R(ωk) is the 2×2 rotation matrix for angle ωk. So PE(pos+k) is a fixed linear function of PE(pos) that depends only on the offset k, never on the absolute position. That means a linear attention projection can, in principle, learn to detect ‘k steps apart’ uniformly everywhere in the sequence. The model gets relative-position sensitivity essentially for free from a purely absolute encoding — a genuinely elegant result, and the direct conceptual ancestor of RoPE.
Absolute vs relative encodings
It helps to name the two philosophies. An absolute encoding gives each position an identity: position 0 gets one vector, position 1 another, and the model reads ‘this token is at slot 7.’ The vanilla sinusoidal scheme and classic learned position embeddings are absolute — the signal is a function of pos alone.
A relative encoding instead injects information about the distance i − j between a query at position i and a key at position j, usually as a bias or modification inside the attention score itself rather than as something added to the input embedding. The intuition is that language often cares about ‘the previous word’ or ‘three tokens back’ far more than about ‘the 512th token in the document.’ The remarkable thing about sinusoids, from the previous section, is that they blur this line: an absolute encoding whose relative structure is linearly recoverable. Modern encodings lean hard into the relative view because it tends to generalize better to sequence lengths not seen in training — a model that only ever learned relationships in terms of distance has less reason to break when the absolute indices run past anything it trained on.
Learned positional embeddings
The other mainstream option drops the hand-designed sinusoids entirely and simply learns a position embedding. You allocate a trainable table P: [N_max, d], one row per position up to a maximum length N_max, initialize it randomly, and let backpropagation shape it — exactly like a word-embedding lookup, but keyed on position instead of token id. The layer input becomes x = E[token] + P[pos]. This is what BERT and GPT-2 use.
The appeal is flexibility: the model is free to discover whatever positional geometry actually helps the task, unconstrained by a designer’s guess about frequencies. In practice learned and sinusoidal encodings perform comparably on sequences within the training range, which is why the original paper reported near-identical results for the two and chose sinusoids partly for a different reason. The cost is parameters — N_max × d extra weights — and, more importantly, a hard ceiling: there is simply no row for position N_max or beyond, so a purely learned absolute table cannot represent a position it never allocated. That single limitation drives the next comparison.
Learned vs sinusoidal: the extrapolation tradeoff
The sharpest practical difference between the two absolute schemes is what happens when a sequence is longer than anything seen in training. A learned table has no entry for out-of-range positions; you must either truncate, interpolate the table, or retrain with a bigger N_max. The sinusoidal formula, by contrast, is defined for every real pos — you can evaluate PE(10000) just as easily as PE(3) — so it extends to longer sequences without any new parameters.
This was an explicit motivation in the original paper: sinusoids were hoped to let the model ‘extrapolate to sequence lengths longer than those encountered during training.’ The reality is more nuanced — naive sinusoidal models often degrade on much longer contexts too, because the attention patterns themselves were only trained on shorter distances even if the encoding is mathematically defined everywhere. But the structural advantage is real, and it is precisely the property that modern relative schemes (RoPE with its interpolation tricks, ALiBi with its distance penalty) work hard to strengthen. Length generalization, not in-distribution accuracy, is where the choice of positional encoding earns its keep.
A short bridge to RoPE
Rotary Position Embedding (RoPE) is the direct heir of the rotation insight, and it now dominates open LLMs (LLaMA, Mistral, Qwen and more). Instead of adding a positional vector to the input, RoPE rotates the query and key vectors themselves, in 2D coordinate pairs, by an angle proportional to the position — reusing the same geometric 10000^(2i/d) frequency schedule for the rotation angles.
The payoff is that the attention score between a query at i and a key at j ends up depending only on their content and on the relative offset i − j, because the two rotations combine into a single rotation by ω(i − j) — exactly the linear-relative property from the sinusoidal section, but now baked structurally into the dot product rather than left for the model to discover. That relative structure is what makes RoPE extend gracefully to longer contexts (often with a simple frequency-interpolation adjustment). RoPE is a sibling of the sinusoidal idea, not a replacement of the concept — it applies the same trigonometry at a smarter place in the pipeline.