POSITIONS · STRETCHING ROTARY EMBEDDINGS
RoPE Extension Strategies
Compute every rotary frequency before and after context extension and count the pairs that would rotate through angles the model never saw in training.
Your experiment
Start with no scaling from 4,096 to 32,768 tokens, base 10,000 and head dimension 128. Count the unseen pairs, then try linear interpolation, the NTK-aware base change, YaRN and Llama 3 scaling, a 500,000 base and a 131,072-token target.
Every input recomputes the result immediately; there is no animation because nothing here unfolds over time. An input outside its allowed range is rejected with a message and the previous valid result stays on screen.
Computed data
Metrics
Frequencies are 1 / base^(2i/d). Linear interpolation divides every frequency by the scale s; the NTK-aware change raises the base to base x s^(d/(d-2)); YaRN and Llama 3 follow transformers' modeling_rope_utils.py (YaRN with beta_fast 32 and beta_slow 1, Llama 3 with low and high frequency factors 1 and 4). A pair counts as unseen when it never completed a full turn within the trained context and the target needs a larger angle than training reached. That geometric test predicts trouble; it does not measure perplexity, and all scaled models still need some long-context fine-tuning.
Wavelengths of a rotary head
Rotary position embedding rotates each pair of dimensions i by an angle position x 1 / base^(2i/d). With base 10,000 and head dimension 128 there are 64 pairs: pair 0 turns once every 6.3 tokens, pair 32 every 628.3, pair 63 every 54,410.1. A model trained on 4,096 tokens has seen pair 0 go round 651.899 times but pair 63 only 0.075 of a turn. Pairs that never completed a turn only know a limited range of angles.
Extrapolation fails on the slow pairs
With Extension method set to none at 32,768 tokens, eight times the trained context, the fast pairs are fine because they already cycled through every angle, but 18 of 64 pairs, from pair 46 on, need angles up to 8x the range seen in training: UNSEEN ANGLES. These pairs carry long-range position information, which is why plain extrapolation degrades. With the RoPE base raised to 500,000 the wavelengths are longer and 32 of 64 pairs are unseen at the same lengths.
Interpolation and the NTK base change
Linear position interpolation divides every frequency by 8, so every angle at 32,768 tokens was already reached at 4,096: ALL ANGLES SEEN. The price is resolution: the fastest pair keeps 12.5% of its angle per token, so neighbouring tokens look more alike. The NTK-aware base change spreads the scaling, leaving fast pairs alone and slow pairs nearly interpolated, but the middle pairs fall short: 17 of 64 pairs stay unseen, up to 1.75x the trained range.
YaRN and Llama 3 scaling
YaRN and Llama 3 split the pairs by wavelength. Pairs that turned many times keep their frequency, pairs longer than the trained context are divided by the full factor, and pairs in between are blended: with Llama 3 scaling pair 40 is divided by 2.301 and pair 48 by 8, and every angle stays inside training. YaRN also multiplies attention logits by 0.1 ln s + 1, 1.2079 for s = 8. Llama 3.1 uses this scaling with base 500,000 to go from 8,192 to 131,072 tokens; there plain extrapolation leaves 29 of 64 pairs unseen.
Reading the tool and its limits
The chart plots log10 wavelength per pair before and after scaling with the trained and target context as flat lines; pairs whose wavelength exceeds the trained line never completed a turn. The table lists every eighth pair. The unseen-angle count is geometry only: it predicts which methods need less fine-tuning, but it cannot tell you the perplexity or retrieval accuracy you will measure at the target length.