Positional encoding is the small piece of arithmetic that lets a transformer know where a token sits in a sequence. Self-attention, on its own, is order-blind: it treats its input as a bag of vectors, so ‘dog bites man’ and ‘man bites dog’ would be indistinguishable to it. That is a fatal problem for language, where order carries meaning, and positional encoding is the fix — a position-dependent signal added to each token embedding before attention ever runs. The original transformer used a fixed pattern of sines and cosines at geometrically spaced frequencies; a beautiful property of that choice is that the relative distance between two positions falls out of a simple linear rotation, which is exactly what attention can learn to exploit. This piece works through the whole idea from first principles: why attention needs position at all, the sinusoidal formula and its 10000^(2i/d) wavelength schedule, the shapes, a worked numeric example, the absolute-versus-relative distinction, learned embeddings, and a short pointer to the modern RoPE and ALiBi siblings.
Why attention is order-blind
Start with the property that makes positional encoding necessary. Self-attention computes, for each token, a weighted average of value vectors, where the weights come from query–key dot products: Attention(X) = softmax(QK^T / √d_k) V. Nowhere in that expression does the index of a token appear. Q, K, and V are produced by the same linear maps applied to every position identically, and the softmax sums over all positions symmetrically.
The precise statement is that self-attention is permutation-equivariant, not permutation-invariant. If P is a permutation matrix that reorders the rows of the input, then Attention(PX) = P·Attention(X) — permute the inputs and the outputs permute in lockstep, but no new information about order enters. The practical consequence is what people loosely call ‘order-blindness’: a given token’s output depends only on which tokens are present (the multiset) and not on their arrangement. So the raw attention block genuinely cannot tell ‘dog bites man’ from ‘man bites dog.’ Recurrent and convolutional networks get order for free from their structure; a transformer has to be told where each token is.
The fix: add position into the embedding
The transformer’s answer is disarmingly simple: manufacture a vector that encodes position and add it to the token embedding before the first attention layer. If E is the token embedding for a word and PE(pos) is the positional vector for its slot, the layer input is x = E + PE(pos). Both live in the same d-dimensional space, so the sum is well defined and the model can learn to read either component.
Why add rather than concatenate? Concatenation would grow the dimension and spend parameters keeping the two signals in separate subspaces. Addition keeps d fixed and lets the network allocate directions in the embedding space to position as it sees fit — in practice the learned W_Q and W_K projections can pull the positional part into whichever subspace is useful for computing attention scores. The key requirement is that PE(pos) be a distinct, structured function of pos: distinct so positions are distinguishable, and structured so that nearby positions get similar vectors and the notion of distance is recoverable. Sinusoids satisfy both.
The sinusoidal formula
The original ‘Attention Is All You Need’ encoding defines each coordinate of the positional vector with a sine or cosine whose frequency depends on the dimension. Index the d coordinates in pairs by i = 0, 1, …, d/2 − 1. Then:
PE(pos, 2i) = sin( pos / 10000^(2i/d) )
PE(pos, 2i+1) = cos( pos / 10000^(2i/d) )So even coordinates get sines, odd coordinates get cosines, and the sine–cosine pair at index i shares a single angular frequency ω_i = 1 / 10000^(2i/d). Equivalently, the argument is pos · ω_i, an angle that advances linearly with position at a rate set by the dimension. At i = 0 the denominator is 10000^0 = 1, giving the highest frequency ω = 1 (a fast oscillation, wavelength 2π). As i grows the denominator grows toward 10000, so the frequency shrinks and the wavelength stretches toward 2π · 10000. Each position therefore maps to a unique vector of coordinates — a bank of ‘clocks’ ticking at many different rates, from very fast to very slow.
The 10000^(2i/d) wavelength schedule
The 10000^(2i/d) term is a geometric progression of wavelengths, and that choice is deliberate. Writing the wavelength as λ_i = 2π · 10000^(2i/d), the wavelengths sweep from 2π (fastest, at i = 0) up to about 2π · 10000 (slowest, at i = d/2 − 1), spaced evenly on a log scale rather than linearly.
This is the same trick as a positional number system or the hands of a clock. The fast, high-frequency dimensions distinguish adjacent positions sharply — they change a lot from pos to pos+1. The slow, low-frequency dimensions barely move between neighbors but disambiguate positions that are far apart, giving the encoding range across long sequences without any two positions colliding. Because the frequencies are geometrically spaced, the encoding carries information at every scale simultaneously — a few tokens apart, a few hundred apart, a few thousand apart. The constant 10000 is a hyperparameter setting the slowest wavelength; it was chosen so the longest period comfortably exceeds realistic sequence lengths, ensuring the slowest clock does not wrap around and alias two distant positions to the same phase.
Shapes: how the encoding is applied
Keeping the tensor shapes straight makes the mechanics concrete. Let N be the sequence length and d (often written d_model) the embedding dimension. The token embeddings form a matrix X: [N, d] (or [B, N, d] with a batch axis B). The positional encoding is a matrix PE: [N, d] whose row pos is the vector PE(pos) from the formula.
Applying it is a single elementwise add, broadcast over the batch: X’ = X + PE, still [N, d]. That is the entire operation — no parameters, no matrix multiply. For the fixed sinusoidal scheme the whole PE matrix can be precomputed once and cached, or even computed on the fly, because it depends only on pos and i, never on the data. Since attention is permutation-equivariant, the add must happen before the first attention layer; do it after, and each layer would still be reordering a bag of position-tagged vectors correctly, but the standard design injects position once at the input and lets it propagate. On the CPU-SLM side this is a negligible cost — an [N, d] add against the O(N^2 d) of attention itself — which is one reason fixed encodings remain attractive for small, tight inference loops.
A worked numeric example
Take a tiny model with d = 4, so i ∈ {0, 1} and there are two frequency pairs. The frequencies are ω_0 = 1 / 10000^(0/4) = 1 and ω_1 = 1 / 10000^(2/4) = 1/100 = 0.01. Coordinates 0 and 1 use ω_0; coordinates 2 and 3 use ω_1.
PE(pos) = [ sin(pos·1), cos(pos·1), sin(pos·0.01), cos(pos·0.01) ]
pos=0: [ sin(0), cos(0), sin(0), cos(0) ] = [ 0.000, 1.000, 0.000, 1.000 ]
pos=1: [ sin(1), cos(1), sin(0.01), cos(0.01) ] = [ 0.841, 0.540, 0.010, 1.000 ]
pos=2: [ sin(2), cos(2), sin(0.02), cos(0.02) ] = [ 0.909, -0.416, 0.020, 1.000 ]Notice the structure. The first pair (the fast clock) swings dramatically between positions — from (0, 1) to (0.841, 0.540) to (0.909, −0.416) — so it sharply separates neighbors. The second pair (the slow clock) barely moves: 0.010 then 0.020, its cosine still essentially 1. Over these three positions the slow clock looks almost constant, but across hundreds of positions it is what keeps distant tokens distinct. Every row is a unique 4-vector, so no two positions collide.
Why sinusoids: relative position is a linear rotation
Here is the property that justifies the whole design. Fix a frequency ω and look at its sine–cosine pair as a 2D vector. Shifting the position by a constant offset k is exactly a rotation of that vector by the angle ωk — and crucially the rotation does not depend on pos. From the angle-addition identities:
sin(ω(pos+k)) = sin(ω·pos)·cos(ωk) + cos(ω·pos)·sin(ωk)
cos(ω(pos+k)) = cos(ω·pos)·cos(ωk) − sin(ω·pos)·sin(ωk)In matrix form, [sin(ω(pos+k)), cos(ω(pos+k))]^T = R(ωk) · [sin(ω·pos), cos(ω·pos)]^T, where R(ωk) is the 2×2 rotation matrix for angle ωk. So PE(pos+k) is a fixed linear function of PE(pos) that depends only on the offset k, never on the absolute position. That means a linear attention projection can, in principle, learn to detect ‘k steps apart’ uniformly everywhere in the sequence. The model gets relative-position sensitivity essentially for free from a purely absolute encoding — a genuinely elegant result, and the direct conceptual ancestor of RoPE.
Absolute vs relative encodings
It helps to name the two philosophies. An absolute encoding gives each position an identity: position 0 gets one vector, position 1 another, and the model reads ‘this token is at slot 7.’ The vanilla sinusoidal scheme and classic learned position embeddings are absolute — the signal is a function of pos alone.
A relative encoding instead injects information about the distance i − j between a query at position i and a key at position j, usually as a bias or modification inside the attention score itself rather than as something added to the input embedding. The intuition is that language often cares about ‘the previous word’ or ‘three tokens back’ far more than about ‘the 512th token in the document.’ The remarkable thing about sinusoids, from the previous section, is that they blur this line: an absolute encoding whose relative structure is linearly recoverable. Modern encodings lean hard into the relative view because it tends to generalize better to sequence lengths not seen in training — a model that only ever learned relationships in terms of distance has less reason to break when the absolute indices run past anything it trained on.
Learned positional embeddings
The other mainstream option drops the hand-designed sinusoids entirely and simply learns a position embedding. You allocate a trainable table P: [N_max, d], one row per position up to a maximum length N_max, initialize it randomly, and let backpropagation shape it — exactly like a word-embedding lookup, but keyed on position instead of token id. The layer input becomes x = E[token] + P[pos]. This is what BERT and GPT-2 use.
The appeal is flexibility: the model is free to discover whatever positional geometry actually helps the task, unconstrained by a designer’s guess about frequencies. In practice learned and sinusoidal encodings perform comparably on sequences within the training range, which is why the original paper reported near-identical results for the two and chose sinusoids partly for a different reason. The cost is parameters — N_max × d extra weights — and, more importantly, a hard ceiling: there is simply no row for position N_max or beyond, so a purely learned absolute table cannot represent a position it never allocated. That single limitation drives the next comparison.
Learned vs sinusoidal: the extrapolation tradeoff
The sharpest practical difference between the two absolute schemes is what happens when a sequence is longer than anything seen in training. A learned table has no entry for out-of-range positions; you must either truncate, interpolate the table, or retrain with a bigger N_max. The sinusoidal formula, by contrast, is defined for every real pos — you can evaluate PE(10000) just as easily as PE(3) — so it extends to longer sequences without any new parameters.
This was an explicit motivation in the original paper: sinusoids were hoped to let the model ‘extrapolate to sequence lengths longer than those encountered during training.’ The reality is more nuanced — naive sinusoidal models often degrade on much longer contexts too, because the attention patterns themselves were only trained on shorter distances even if the encoding is mathematically defined everywhere. But the structural advantage is real, and it is precisely the property that modern relative schemes (RoPE with its interpolation tricks, ALiBi with its distance penalty) work hard to strengthen. Length generalization, not in-distribution accuracy, is where the choice of positional encoding earns its keep.
A short bridge to RoPE
Rotary Position Embedding (RoPE) is the direct heir of the rotation insight, and it now dominates open LLMs (LLaMA, Mistral, Qwen and more). Instead of adding a positional vector to the input, RoPE rotates the query and key vectors themselves, in 2D coordinate pairs, by an angle proportional to the position — reusing the same geometric 10000^(2i/d) frequency schedule for the rotation angles.
The payoff is that the attention score between a query at i and a key at j ends up depending only on their content and on the relative offset i − j, because the two rotations combine into a single rotation by ω(i − j) — exactly the linear-relative property from the sinusoidal section, but now baked structurally into the dot product rather than left for the model to discover. That relative structure is what makes RoPE extend gracefully to longer contexts (often with a simple frequency-interpolation adjustment). RoPE is a sibling of the sinusoidal idea, not a replacement of the concept — it applies the same trigonometry at a smarter place in the pipeline.
A short bridge to ALiBi
ALiBi (Attention with Linear Biases) takes an even simpler route and skips positional vectors altogether. It adds no encoding to the embeddings; instead it adds a static, non-learned penalty to each attention score that grows linearly with the distance between query and key: a query attending to a key k positions away has its pre-softmax score reduced by m · k, where m is a fixed per-head slope.
The effect is a built-in recency bias — nearer tokens are favored, farther ones gently discouraged — that is purely a function of relative distance and involves no trained parameters at all. ALiBi’s headline claim is strong length generalization: because the bias is just a distance rule with no learned table and no wrapping frequency, a model trained on short sequences continues to behave sensibly on much longer ones. Alongside RoPE it rounds out the modern picture: the field moved from ‘add an absolute vector’ toward ‘encode relative distance directly in the attention scores.’ Both are worth a full treatment of their own; here they are signposts showing where the sinusoidal idea leads.
Practical implications for CPU-SLM inference
For small language models running on a CPU, the positional-encoding choice has concrete engineering consequences beyond accuracy. A fixed sinusoidal encoding costs no parameters and no learned state: the PE: [N, d] matrix is either precomputed once at load time and cached, or generated cheaply on the fly, and the runtime cost is a single [N, d] elementwise add — utterly negligible next to the O(N^2 d) attention and the O(N d^2) feed-forward work that dominate a forward pass.
A learned table costs N_max × d weights that ship in the model file and occupy memory, which matters when you are squeezing a model into a tight footprint. RoPE, meanwhile, adds a small per-token rotation applied to Q and K inside each attention layer — still cheap, and it composes cleanly with a KV cache because each cached key was already rotated by its own position when it was written. When you extend a CPU-hosted SLM to longer contexts than it trained on, the encoding is often the first thing that breaks: a learned table simply has no rows, and that failure is silent unless you guard for it. Knowing which scheme your model uses tells you exactly what will happen at the length boundary.
Common pitfalls
A handful of mistakes recur. Forgetting the encoding entirely is the classic one: a from-scratch transformer that trains but never learns word order, because the positional add was omitted and attention is quietly treating every input as a bag of tokens. Adding position in the wrong place — after attention instead of before it, or only in some layers — produces subtler degradations. Mismatched N_max with a learned table causes index errors or silent truncation the moment a real input exceeds the trained maximum.
Two more are conceptual. Assuming sinusoidal encodings freely extrapolate to any length overpromises: the formula is defined everywhere, but the trained attention behavior is not, so very long contexts can still fall apart without extra care. And conflating the schemes — expecting RoPE and additive sinusoids to be swappable — leads to bugs, since RoPE rotates Q and K inside attention while sinusoids are added to the input; they are not interchangeable drop-ins. The safe habit is to know precisely which encoding a model uses, where it is applied, and what its behavior is at and beyond the maximum trained length.
10000^(2i/d) schedule: fast clocks separate neighbors, slow clocks disambiguate distant positions. The elegant part is that shifting position by a constant offset k is just a fixed rotation of each sine–cosine pair, so relative distance is linearly recoverable from a purely absolute encoding — the seed that grows into RoPE. Learned position embeddings trade a parameter table for flexibility but hit a hard ceiling at their maximum length, whereas sinusoids are defined for every position and extend more gracefully. The modern siblings, RoPE and ALiBi, push further into the relative view — rotating queries and keys, or penalizing attention by distance — because length generalization, not in-range accuracy, is where positional encoding truly earns its keep.