Positional encoding: addition versus rotation

Set vectors and positions

First coordinate pair

Blue: first vector. Amber: second vector. Dashed: original. Solid: encoded. The horizontal axis is coordinate 0 and the vertical axis is coordinate 1. Tables expose all values, including overlapping arrows.

Complete vector coordinates

PE columns show the additive sine/cosine encoding even in RoPE mode; RoPE outputs use rotation, not these additive values.

Two operations, two meanings

Choose RoPE to treat the editable arrays as query and key vectors after projection. Adjacent coordinates form two-dimensional planes. In each plane the controller applies a genuine rotation with angle position times base raised to minus twice the pair index divided by the vector dimension. Choose sinusoidal addition to treat the arrays as token embeddings instead. Alternating sine and cosine coordinates are added before projection. The displayed dot then compares those encoded embeddings; it is not an attention score from a trained network. No learned projection, softmax or value aggregation is performed here.

Edit every input and inspect every coordinate

First vector and Second vector accept JSON arrays with equal even length between two and sixteen. Changing their length changes the frequency schedule and the number of rows. Components must be finite numbers between minus one hundred and one hundred. First position and Second position accept nonnegative integer token indices up to one million. Frequency base accepts values from two to one million and defaults to ten thousand. A larger base slows the higher-index pairs; the first pair always has frequency one radian per token. Invalid inputs clear the computed tables and drawing instead of leaving stale results.

Test relative position with exact experiments

Reset defaults selects RoPE, two identical four-component vectors [1,0,1,0], positions zero and one, and base ten thousand. The encoded dot is cos(1) + cos(0.01), approximately 1.540252. Each vector retains norm square root of two. Shift both positions +10: each visible vector rotates, while the complete dot stays the same within floating-point error. Now change only Second position to twelve. The relative offset becomes two and the dot becomes cos(2) + cos(0.02), approximately 0.583653. Set both positions equal: the encoded dot returns to the original value two.

Compare addition and rotation without a timing illusion

Switch to sinusoidal addition and set both positions to zero. Each [1,0,1,0] embedding becomes [1,1,1,1], so its squared norm and the encoded dot are four. A shared position shift generally changes this embedding dot because content-position cross terms remain. The blue and amber arrows show only the first coordinate pair; dashed arrows are inputs and solid arrows are outputs. Equal axis scales preserve angles, while automatic zoom accommodates the inputs. The coordinate table is the complete text alternative and the pair table shows every dot-product contribution. This is a direct geometry tool: changes are immediate and remain inspectable indefinitely. Reset defaults restores every input.

Boundaries and what this does not establish

A zero vector remains zero under RoPE, but sinusoidal addition introduces a nonzero position vector. Odd dimensions, unequal array lengths, missing positions and malformed JSON are rejected explicitly. Rotations preserve norms and make the query-key dot depend on relative offset when content vectors are fixed. They do not prove that a trained model will generalize to arbitrary unseen context lengths. Scores can oscillate with distance; this experiment does not promise monotonic decay, language quality, or universal architecture adoption. Calculations use JavaScript double precision and adjacent-pair convention; implementations that arrange paired coordinates differently need the corresponding permutation.