Why it matters

Long context is hot. Understanding shapes context extension.

Advertisement

The architecture

NTK-aware: interpolate high frequencies more than low.

YaRN: refined NTK with temperature.

Context extensionRoPE base freqoriginalNTK / YaRNscale non-uniformlyExtended context8k → 32k+YaRN better preserves quality than linear interpolation
Context scaling.
Advertisement

How it works end to end

Linear PI: scale all frequencies equally (bad).

NTK-aware: scale high freq less.

YaRN: NTK + temperature + fine-tune.