Why it matters
RoPE is the modern positional encoding standard. Understanding it matters for extending context length, fine-tuning models, and debugging position-related issues.
The architecture
Split Q, K into pairs of dimensions. Rotate each pair by an angle θ_i × m for position m, where θ_i are frequencies from 1 down to a base (typically 10000).
Dot product Q·K after rotation depends on the position difference, not absolute positions.
How it works end to end
Frequencies: high-frequency components capture local structure; low-frequency captures long-range. Base of 10000 spreads them across scales.
Context extension: NTK-aware scaling adjusts base to extrapolate to longer contexts. YaRN, LongRoPE further improve extrapolation.
Advantages over absolute PE: no separate embedding table, encodes relative position naturally, works with attention algebra.