RoPE and Long Context: Why Models Fail Past Their Trained Length

How RoPE turns position into rotation

RoPE, introduced with RoFormer, encodes position by rotating query and key vectors instead of adding a position vector to the input. The head dimension d is split into d/2 pairs. Pair i is treated as a 2-D vector and rotated by the angle m x theta_i at position m, where theta_i = base^(-2i/d) and the base is traditionally 10,000. Because a rotation of the query by m theta and of the key by n theta leaves a dot product that depends only on (m - n) theta, attention scores depend on relative position, which is the property that made RoPE popular.

The frequencies span a huge range. With d = 128 and base 10,000:

theta_i = 10000^(-2i/128), i = 0 .. 63

pair  0: theta = 1.0 rad/token        wavelength =      6.3 tokens
pair 32: theta = 0.01 rad/token       wavelength =    628 tokens
pair 46: theta = 1.3e-3 rad/token     wavelength =  4,700 tokens
pair 63: theta = 1.15e-4 rad/token    wavelength = 54,600 tokens

pairs whose wavelength exceeds 4,096 tokens: i = 46 .. 63 (18 of 64)

Fast pairs complete thousands of turns over a 4,096-token training context, so during training they meet every possible angle many times. Slow pairs do not complete even one turn. Pair 63 rotates only about 0.47 radians across the whole 4,096-token window. Those slow pairs are what the model uses to judge long distances, and they are where extension goes wrong.

Why models fail past their trained length

Serve the same model at 16,384 tokens without any change and pair 63 now rotates through about 1.9 radians: four times the largest angle it ever saw. The other 17 slow pairs are in the same position. The weights that read those dimensions were fitted to a narrow arc of angles and have no reason to behave sensibly outside it. The Position Interpolation paper by Chen and colleagues showed that attention scores under such extrapolation can become very large and erratic, while interpolated positions stay well behaved, which matches what practitioners see: perplexity rises sharply just past the trained length, and retrieval of a fact from early in the context fails.

Two details make this worse in production. First, the failure is not graceful. In the lab, passkey retrieval drops from 100% at 4k to near zero at 8k for an unscaled model, because random logits from out-of-range pairs let distractor tokens beat the real answer. Second, it can be silent. Output stays grammatical because local structure is carried by fast pairs that are still in range; only long-range use of the context breaks. Teams that test with short prompts or ask summary-style questions can ship this without noticing.

Three ways to remap positions

Every RoPE extension method changes the frequencies so that the longest serving position produces angles the model has seen, or nearly seen. They differ in which pairs they touch.

• Linear position interpolation (PI). Divide every position by the scale factor s, which is the same as dividing every theta_i by s. A 4k model with s = 4 maps 16k positions back into the 0 to 4k range. Nothing is out of range, but fast pairs are compressed too: neighbouring tokens that used to differ by 1 radian now differ by 0.25, so fine local distinctions blur until the model is fine-tuned to the new spacing. The PI paper extended LLaMA models to 32k with about 1,000 fine-tuning steps.
• NTK-aware scaling. Raise the base instead: base' = base x s^(d/(d-2)). This leaves the fastest pairs almost untouched and divides the slowest pair's frequency by roughly s, with a smooth progression in between. It works reasonably without fine-tuning, but the middle-to-slow pairs are not compressed quite enough, so in practice the factor must be set above the actual extension ratio. A dynamic variant raises the factor only as the current sequence grows.
• YaRN. Treat pairs by wavelength. Pairs that complete many turns in the trained window are left alone, pairs whose wavelength exceeds the window are fully interpolated by s, and a linear ramp blends the two (the paper uses ramp bounds of 1 and 32 turns for LLaMA). YaRN also scales attention logits by a small temperature, roughly 0.1 ln(s) + 1, to counter the flattening of attention at long lengths. The authors report reaching long contexts with considerably less fine-tuning than earlier methods.

In the lab the left panel shows each of the 64 pairs: navy bars have seen every angle, teal bars are inside the trained range, and red bars are past it. Switch from no scaling to PI, NTK and YaRN and watch which bars move.

Fine-tuning: what the extra steps buy

Scaling without training is a remapping trick; fine-tuning on long sequences teaches the model the new geometry. Two things improve. The model adapts to compressed spacing, recovering short-range quality lost to interpolation, and it learns to use positions out to the fine-tuning length. Fine-tuning at one length does not teach angles beyond it: a model fine-tuned at 32k with s = 8 still breaks at 64k.

Plain fine-tuning without any scaling is the slowest option, because the slow pairs need to learn entirely new angle ranges. The PI paper compared it directly and found that extending by direct fine-tuning progressed very slowly, while interpolation reached the target window within its fine-tuning budget. The cost of each step also grows with length: attention cost is quadratic in sequence length, so a step at 128k is much more expensive than a step at 32k. In the lab, an illustrative price grows linearly with target length, which understates the real growth.

The data matters as much as the step count. Long-context fine-tuning needs sequences that genuinely use their length, such as whole books, long code files or concatenated related documents, rather than short texts packed together, which teach the model that distant tokens are irrelevant. Mixing in ordinary short data helps protect short-prompt behaviour. Treat the fine-tuning set as a product decision: if your workload is contracts, include long contracts or similar long structured documents.

Turning it on in real systems

Most serving stacks read the scaling settings from the model config. In Hugging Face Transformers, the rope_scaling dictionary selects the method and factor, and rope_theta sets the base. A YaRN extension of a 32k model to roughly 128k looks like this:

{
  "max_position_embeddings": 131072,
  "rope_theta": 1000000.0,
  "rope_scaling": {
    "rope_type": "yarn",
    "factor": 4.0,
    "original_max_position_embeddings": 32768
  }
}

# vLLM: the same can be passed at launch (flag names vary by version)
vllm serve my-model --max-model-len 131072 \
  --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}'

Older configs use the key type instead of rope_type; supported values include linear, dynamic and yarn, and some model families ship their own variant, for example Llama 3.1's llama3 type, which interpolates low-frequency pairs and leaves high-frequency pairs alone. Newer vLLM versions may expect these overrides through a general config-override argument instead of a dedicated flag. Check your version's documentation, and check the model card: many long-context models already ship with scaling configured, and adding a second factor on top double-scales positions.

Note that most servers apply static scaling to every request, so a short prompt is also processed with compressed frequencies. Some model cards warn about exactly this and suggest enabling the long-context setting only on deployments that need it.

The serving bill grows with every token

Even when quality holds, long context is not free. KV cache grows linearly with tokens; prefill attention grows quadratically. For an 8B-class model with eight KV heads (128 KiB of cache per token in FP16), the lab's illustrative numbers are:

The arithmetic behind the prefill column is weights work of 2 x parameters per token plus attention work of roughly 2 x L^2 x hidden size x layers for the causal score and value products, divided by sustained throughput. At short lengths the weight term dominates; past about 32k the quadratic term takes over. This is why extending to 128k when documents never exceed 32k is a costly mistake: it raises fine-tuning cost, cache reservation and worst-case latency for no benefit. Mission four in the lab asks you to undo exactly that over-extension.

Testing effective context honestly

A configured window is a claim; the effective context is what the model can use. Benchmarks such as RULER found that many models claiming long windows perform well on simple needle retrieval but degrade on harder tasks such as multi-hop tracing and aggregation well before their advertised length. A credible acceptance test does four things:

1. Runs retrieval at several lengths, not one, so you see where accuracy falls off.
2. Places the target fact at several depths, including the oldest part of the context, because depth and length interact.
3. Includes at least one task that needs more than locating a string, such as combining two facts.
4. Re-runs short-context evaluations, because scaling can quietly damage them.

The lab's chart is a miniature version: passkey accuracy by context length, with markers for the trained and serving lengths. Its effective-context tile reports the longest length up to which every tested length stays at or above 90%. The facts at the far end event moves the needle into the oldest 15% of the context, which exposes weak interpolation that a uniform test hides. Mission three asks you to pass that test at 128k on a fine-tuning budget.

A worked example: Belgavi's Book Store contracts

Suppose the hypothetical Belgavi's Book Store wants an assistant that answers questions about supplier contracts of up to 30,000 tokens, using an open model trained at 4,096 tokens. The arithmetic says 18 of 64 pairs are slow enough to break, so serving at 32k needs a scale factor of at least 8. With YaRN at s = 8 and no fine-tuning, the lab's synthetic passkey test passes at 32k with short-context quality near the original, but accuracy at intermediate lengths wobbles. Adding 50 fine-tuning steps at 32k, at the illustrative $3.20 per step, steadies every length for about $160.

The same team could have chosen 128k "for headroom". The lab prices that at roughly 33 seconds of prefill per maximal request, 17 GB of cache per request, and $12.80 per fine-tuning step. Nothing in the contract workload uses that headroom. The disciplined choice is the smallest window that covers the real document distribution plus a margin, with an acceptance test at that length, and a separate deployment if a genuinely long-context use case appears.

Failure modes and common mistakes

• Raising the window without scaling. The server accepts longer prompts; the model does not use them. Detect it with retrieval tests past the trained length, not with perplexity on short text.
• Double scaling. Applying a factor to a checkpoint that already ships with one compresses positions twice. Read the model's config before adding overrides.
• Context creep. Clients send longer prompts than the length you scaled for, and quality falls off a cliff at exactly trained length x factor. Enforce the limit at the gateway and alert on prompt-length percentiles. Mission two in the lab turns this event on and asks you to recover without fine-tuning.
• Short-context regressions. Linear interpolation without enough fine-tuning blurs local order. Keep short-prompt evaluations in every long-context release.
• Confusing position failure with lost-in-the-middle. A model can be inside its trained range and still under-use facts in the middle of a long prompt. That is an attention-allocation problem, fixed by ordering and retrieval, not by RoPE scaling.

Run the lab: Stretch the context: RoPE scaling without breaking retrieval

The lab models one attention head of dimension 128 with standard RoPE frequencies. The left panel compares each pair's largest angle at the serving length with the largest angle it saw in training; three dials show a fast, a middle and a slow pair rotating. The right panel runs a synthetic passkey test at seven context lengths. Start with mission one, which serves a 4k model at 16k with no scaling, then try to keep retrieval intact while clients, budgets and documents change.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Read max_position_embeddings, rope_theta and any rope_scaling block from each deployed model's config, and record the original trained length.
2. Compute how many RoPE pairs have a wavelength longer than the trained context, and the smallest scale factor that covers your serving length.
3. Run a passkey or needle test at four or more lengths and three depths, plus one multi-fact task, before raising any context limit.
4. Re-run your short-prompt evaluations after enabling scaling to catch interpolation damage.
5. Enforce the maximum prompt length at the gateway and alert on p95 and p99 prompt tokens.
6. Size the window from your real document distribution and price the KV cache and prefill cost at that length before choosing a bigger one.

Sources and further reading

• RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al.)
• Extending Context Window of Large Language Models via Positional Interpolation (Chen et al., 2023)
• YaRN: Efficient Context Window Extension of Large Language Models (Peng et al., 2023)
• RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al., 2024)
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (Ding et al., 2024)
• vLLM documentation: engine arguments

Related in this series:

• Grouped-Query Attention: How Sharing KV Heads Shrinks Inference Memory
• KV Cache Capacity Planning: How Many Concurrent Users Fit on One GPU
• Lost in the Middle: Ordering and Packing Retrieved Context for Long Prompts

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.