Evidence label: deterministic synthetic model. This page runs a small JavaScript model of RoPE context extension. It computes the real rotation frequencies for a 128-dimension head under each scaling method, works out which pairs exceed the angles seen in training, and then runs a synthetic passkey test: a matching key competes with forty distractors at random positions, and pairs rotating past their learned range add noise to the logits. Interpolation without fine-tuning adds extra noise, and compression of fast pairs reduces a short-context quality proxy. Nothing here runs a real model; all numbers are calculated estimates.
Play the lab
How to play
Pick a scaling method and factor, the model's trained context, the serving context and how many fine-tuning steps to spend. Every change re-runs the same seeded test, so differences come from your change. The tiles show passkey accuracy at the serving length, the effective context, a short-context quality proxy, how many pairs are past their trained angles, an illustrative fine-tuning cost, and the prefill time and cache size at the serving length. Events double the prompt length clients actually send, or push the fact to the oldest part of the context.
Missions unlock in order; each loads a broken setup. Stars reward answers that keep short-context quality high or spend less on fine-tuning.
What the model assumes
Frequencies follow theta_i = 10,000^(-2i/128). NTK scaling raises the base by s^(128/126); YaRN uses a linear ramp between 1 and 32 rotations over the trained length and a logit temperature of 1 + 0.1 ln(s). A pair that completed a full turn in training is treated as safe at any length. Unseen rotation, capped at pi per pair, becomes Gaussian logit noise. Fine-tuning moves each pair's learned range toward the serving length exponentially with steps, faster for interpolated methods than for raw extrapolation. Fine-tuning costs an illustrative $0.10 per step per 1,024 tokens of target length; prefill assumes an 8B-class model at 200 TFLOP/s effective.
What it cannot tell you
Real models do not fail through Gaussian noise; they fail through learned weights meeting inputs outside their training distribution, and the exact shape of the cliff depends on the model and data. Passkey retrieval is the easiest long-context task, so a pass here says little about multi-hop reasoning. Use the lab to understand which pairs break and why each method helps, then test your own model at your own lengths.
Read the full explanation
- RoPE and Long Context: Why Models Fail Past Their Trained Length (companion article)
- Grouped-Query Attention: How Sharing KV Heads Shrinks Inference Memory
- KV Cache Capacity Planning: How Many Concurrent Users Fit on One GPU
All AI in Production labs · Lab catalog
- RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al.)
- Extending Context Window of Large Language Models via Positional Interpolation (Chen et al., 2023)
- YaRN: Efficient Context Window Extension of Large Language Models (Peng et al., 2023)
- RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al., 2024)
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (Ding et al., 2024)
- vLLM documentation: engine arguments
Views are my own and do not represent any employer, past or present.