Why it matters

Wrong context sizing causes truncated outputs or failed calls. Understanding limits guides application design.

Advertisement

The architecture

Context = input prompt + generated output. Both consume the context budget.

Attention is O(n²) in context; long context is expensive to compute.

Context windowInput tokensprompt + historyOutput tokensgenerationTotal ≤ context limitbudgetModern models: 128K standard; 1M+ available; retrieval augments beyond even that
Context budgeting.
Advertisement

How it works end to end

Attention memory scales quadratically. Kv cache scales linearly. Both grow fast with long context.

Extension techniques: rope scaling (NTK, YaRN), retrieval augmentation (RAG), summarization of old context.

Quality often degrades on very long context: 'lost in the middle' — models attend more to beginning and end than middle.