Why architecture matters here
The architecture matters because the failure it prevents is not a crash but a subtle degradation of reasoning. When a window overflows and history is truncated, the agent does not announce that it forgot something — it simply proceeds as if the dropped context never existed, confidently pursuing a goal it half-remembers or repeating a tool call whose result scrolled off. These are the hardest bugs to diagnose because the model's output is fluent and plausible; only by comparing against what the user actually said does the loss become visible. Compaction exists to keep the essential content in the window so the agent's reasoning stays grounded in the full history, not an accidental suffix of it.
Cost is the second driver. In most agent runtimes the entire context is re-sent to the model on every step, so a window that grows linearly with the conversation makes each turn more expensive than the last — a long session pays quadratically in tokens overall. Compaction bends that curve: by holding the window near a stable size, per-turn cost plateaus instead of climbing, which is the difference between an agent that is economical over hundreds of turns and one that becomes prohibitively expensive after a few dozen.
There is also a quality argument that is easy to miss: a smaller, well-curated window often reasons better than a huge one. Models attend less reliably over very long contexts — the middle of a long prompt is where relevant facts get lost — so a window packed with stale tool output can bury the signal the agent needs. Compaction is therefore not merely a cost or capacity measure; it is a relevance filter that keeps the prompt dense with what matters, which can improve accuracy even when the raw history would still have fit.
It is worth naming who bears the cost when compaction is absent, because it clarifies the stakes. The user bears it as an agent that mysteriously forgets a preference stated ten minutes ago, or that re-asks for information already given — an experience that reads as incompetence even though the model is capable. The operator bears it as a bill that grows super-linearly with session length, since each turn re-sends a larger prompt, and as latency that creeps up turn over turn as the window swells. The model bears it as attention diluted across a wall of stale tool output, so the one relevant fact is buried in the middle where long-context recall is weakest. Compaction addresses all three at their common root — an unbounded window — rather than papering over the symptoms. Consider a coding agent that spent turns three through fifteen exploring a repository: those turns are full of file contents it has already extracted the relevant functions from, so their raw text is pure ballast, while the conclusions drawn from them — this module owns auth, that one has the bug — are the gold. Compaction is precisely the operation that keeps the conclusions and discards the ballast, and doing it deliberately is what lets such an agent work a problem across fifty turns instead of drowning in its own exploration by turn twenty.
Finally, compaction is where an agent's memory policy becomes explicit and controllable. The choices it forces — what is essential enough to pin, what is safe to summarize, what can be evicted to external storage and retrieved on demand — are the same choices a thoughtful notetaker makes. Encoding them in a deliberate architecture means an operator can reason about, test, and tune what the agent remembers, rather than leaving it to the accident of which messages happened to fit. That transparency is what turns 'the agent sometimes forgets things' from an inscrutable flakiness into a policy you can inspect and improve.
The architecture: every piece explained
The token meter is the sensor. It measures the current window's size against the model's limit — or, more usefully, against a working budget set below the limit to leave headroom for the next turn's output and tool results. Because token counts depend on the tokenizer, the meter counts real tokens, not characters, and it runs before each step so the decision to compact is made with an accurate picture of how full the window is.
The compaction trigger fires when the meter crosses a threshold — commonly a fraction of the budget such as seventy-five percent — rather than waiting for the hard limit. Triggering early is deliberate: it leaves room to run the summarizer and to absorb the next turn without overflowing mid-step. Some designs also trigger on turn count or on time, but the token threshold is the primary signal because it maps directly to the constraint being managed.
The segment selector decides what to compact. The system prompt is always kept verbatim; the most recent turns are kept verbatim because they are the live working set; and the middle — old turns and their tool results — is the eligible segment. The selector typically keeps the last N turns intact and marks everything older as a candidate, so compaction always compresses the settled past and never the active present. The summarizer then condenses that segment: a model call (often a cheaper model) reads the old turns and produces a compact summary capturing goals, decisions, key facts, and unresolved threads. The summary replaces the segment in the window.
The preservation policy is what keeps compaction safe. Beyond the recent-turns rule, it defines pinned facts — items that must survive verbatim no matter how old, such as the user's stated objective, hard constraints, credentials or identifiers the task depends on, and the current plan or todo state. Pins are excluded from summarization and always re-inserted. External memory is the escape hatch for detail that is too bulky to keep but too valuable to lose: the full text of an evicted segment is written to a retrievable store keyed so the agent can fetch it later if a question demands the specifics the summary elided. Together these pieces mean compaction is not lossy truncation but a tiered memory: hot context in the window, warm summary of the recent past, and cold detail offloaded but recallable — which is exactly the layering that lets a session run indefinitely without either overflowing or forgetting what mattered.
End-to-end flow
Trace a long research session. For the first dozen turns the window sits comfortably below budget and the meter never trips; every message is kept verbatim and the agent reasons over full history. As the user drills into sources, each turn appends a large tool result — a fetched page, a long search payload — and the window climbs. Before the twentieth step, the meter reports the window has crossed seventy-five percent of budget, and the compaction trigger fires.
The segment selector keeps the system prompt and the last three turns verbatim and marks turns four through sixteen as the eligible segment. It also gathers the pinned facts — the user's original research question and two constraints they stated early. The summarizer reads the eligible segment and produces a compact paragraph: what was searched, which sources proved relevant, what was ruled out, and the two open questions still being pursued. The full text of those old turns is written to external memory under keys the agent can query later.
The runtime then rebuilds the window: system prompt, the pinned question and constraints, the summary of the settled past, and the three most recent turns verbatim. The window is now well under budget again, and the next step proceeds with the thread of the investigation intact but the bulk of the old tool output compressed away. If a later turn needs a specific detail the summary dropped — an exact figure from source five — the agent issues a memory retrieval keyed to that source and pulls the full text back into context just for that step.
Watch how the tiers cooperate across the rest of the session. Each subsequent overflow compacts the newly-settled turns into the existing summary — the summary itself is periodically re-summarized so it, too, stays bounded — while pins ride through every compaction untouched and recent turns rotate through verbatim. The window's size oscillates in a stable band rather than climbing, per-turn cost plateaus, and the agent retains its goal, its constraints, and its recent working set at all times. The detail it chose not to keep is not gone but parked in external memory, one retrieval away. The end-to-end flow, read carefully, is a memory hierarchy in miniature — and it is that layering, not any single summary, that lets the session run far past the raw context limit without losing the thread.