Why it matters
Positional encoding is where transformer flexibility for arbitrary sequences comes from. Getting it right (or wrong) directly affects context handling.
The architecture
Sinusoidal positional encoding (original): fixed sine and cosine waves at different frequencies. Position i gets encoding based on i and dimension index.
Learned positional encoding: a learnable embedding per position. Simpler but doesn't extrapolate to unseen positions.
How it works end to end
Rotary Position Embedding (RoPE): rotates Q and K by an angle proportional to position. Dot product Q·K then depends on relative position. Standard in modern LLMs (LLaMA, GPT-NeoX).
ALiBi: adds a linear bias to attention scores based on position distance. Very simple, extrapolates well.
Extrapolation: extending context beyond training length. Learned PE fails; sinusoidal and RoPE work with modifications.