Why architecture matters here
The architecture matters because compaction is a lossy operation on the agent's only memory, and lossy operations on memory produce failures that are extremely hard to debug. When an agent behaves incorrectly because a fact was compacted away, the symptom is not an error — it is a plausible, confident response that happens to be wrong, and the evidence of why is gone by construction. You cannot look at the context and see the missing piece; you can only see what remains.
This makes the strategy choice consequential in a way that most performance optimizations are not. Consider the difference between dropping tool payloads and summarizing turns. Dropping payloads is nearly free and nearly safe: the agent already extracted what it needed from that JSON blob when it saw it, and its own response records the conclusion. The payload is recoverable too — the tool can be called again if the agent genuinely needs it. Summarizing turns, by contrast, is an irreversible lossy transform performed by a model that does not know which detail will matter three turns from now. The summary will be fluent and reasonable and will have silently dropped the constraint the user mentioned in passing that the whole task depends on.
The cost dimension pushes hard against the safety dimension, which is why this is a genuine trade rather than a best practice you can just adopt. Every turn re-sends the entire context, so length is not a one-time cost but a per-turn cost that grows with the conversation. A fifty-turn conversation does not cost fifty units; it costs closer to the sum of an arithmetic series, because turn fifty pays for all forty-nine before it. Compaction bends that curve from quadratic toward linear — the difference between a viable product and one whose unit economics collapse as engagement grows, which is a cruel failure mode: your best users cost you the most.
Prompt caching complicates the calculus rather than simplifying it. With caching, the marginal cost of a long stable prefix is much lower than the sticker price, which weakens the argument for aggressive compaction considerably. The optimal policy is therefore not 'compact as much as possible' but 'compact as rarely as possible, as much as necessary' — let the context grow under a warm cache, and when you must compact, compact hard enough that you will not need to again for many turns. Frequent small compactions are the worst of both worlds: you pay the cache invalidation repeatedly and you pay the summarization model repeatedly, for a saving you immediately give back.
Finally, compaction has to be a first-class part of the session model rather than a filter applied at request time. If you compact on the way to the model without writing the result back, you recompute the same summary on every turn — paying for a model call to produce an artifact you already produced — and worse, the summary may differ slightly each time, so your context is non-deterministic even when the history is not. Storing the compaction back into the session as a first-class event makes it computed once, auditable, and stable, which matters enormously when you are trying to reproduce a bad interaction.
The architecture: every piece explained
The token budgeter is the sensor. Before every model call, the assembled context is measured — not estimated from character counts, but tokenized with the same tokenizer the target model uses, because the difference between an estimate and a count is exactly the margin that makes an overflow error surprising. The budgeter compares against a threshold that is deliberately below the window limit, because you need headroom for the model's own response and for the possibility that the next turn adds a large tool result. Triggering at 100% of the window means you overflow on the turn after the one you measured.
The pinned set is the safety mechanism, and it is what separates compaction from truncation. Certain events are never evicted under any strategy: the system instruction, which defines the agent's behaviour; the original task goal, which defines what success means; and the most recent N turns verbatim, which carry the immediate conversational context. Pinning is declarative — these events are marked, and every strategy respects the mark. Without a pinned set, the first thing a naive front-truncation removes is the system prompt, because it is the oldest event in the log, and the agent immediately forgets its own instructions in the least debuggable way possible.
The strategies form a ladder from cheap and safe to expensive and lossy, and a good implementation tries them in order. Tool payload elision is the first rung: replace a large tool result with a compact reference — the tool name, its arguments, and a short digest of the result — keeping the fact of the call while discarding the bulk. This is often enough on its own, because tool results dominate context in most agentic workloads by a wide margin. The second rung is relevance-based eviction: drop events with low relevance to the current task, using embedding similarity or simple recency heuristics. The third rung is summarization: hand a stretch of old turns to a model and replace them with its summary. This is the most powerful and the most dangerous, and it should be the last resort rather than the default.
The summarization step deserves care because it is where the losses concentrate. The prompt matters enormously: 'summarize this conversation' produces a narrative summary optimized for a human reader, which is not what you want. What you want is a summary optimized for an agent to continue working from — decisions made, constraints established, facts discovered, open questions, current state. Structuring the summary explicitly around those categories, rather than letting the model write prose, retains dramatically more of what actually matters. It is also worth summarizing into a schema rather than free text, so downstream turns can reason about the parts rather than re-reading a paragraph.
The write-back path closes the loop. The compacted result becomes a new event in the session — a compaction event that records what was compacted, what strategy was used, and what the resulting summary is. The original events are retained in the session store for audit and debugging but are marked as superseded, so context assembly skips them. This gives you three properties that matter: the summary is computed once rather than per turn, the compaction is auditable when someone asks why the agent forgot something, and the raw history is still there if you need to reconstruct what actually happened. The context sent to the model is then: pinned events, plus compaction summaries in order, plus the verbatim recent tail.
End-to-end flow
Follow a session from turn one to turn forty. Early on there is nothing to do: the budgeter measures each assembled context, finds it far under threshold, and passes it through untouched. The context grows monotonically, the prefix stays stable, and prompt caching makes each turn cheap because only the newly appended tail is uncached.
Around turn eighteen, the agent has made a dozen tool calls, several returning substantial documents. The assembled context crosses the compaction threshold — say 70% of the window, leaving comfortable headroom for a response and one more large tool result. The budgeter signals, and strategy selection begins with the cheapest rung.
Tool payload elision runs first. The system walks the event log, finds tool result events older than the pinned recent tail, and replaces their payloads with digests: the tool name, the arguments, and a one-line description of what was returned. Three documents totalling 12,000 tokens become three references totalling 200. The budgeter re-measures: the context is now at 41% of the window, comfortably under threshold. No summarization is needed, no model call is made, and nothing that the agent reasoned over was lost — because whatever it concluded from those documents is recorded verbatim in its own responses, which were not touched. Compaction is complete. The cost is one cache invalidation, since the middle of the context changed.
By turn thirty-four the context has grown again, this time from conversational turns rather than tool payloads, and elision has nothing left to elide. The threshold trips and the ladder descends to summarization. The system selects the span from turn one to turn twenty — old enough to be outside the verbatim tail, excluding the pinned system prompt and task goal — and sends it to a model with a structured summarization prompt. The result is not prose but a structured object: decisions made, constraints established, facts discovered, open questions. This becomes a compaction event written back to the session, superseding turns one through twenty. Assembly now produces: system prompt (pinned), task goal (pinned), the structured summary of turns 1-20, verbatim turns 21-34.
Turn thirty-five proceeds normally. The context is now well under threshold, and critically, the prefix is stable again — the summary does not change, so caching works from turn thirty-five onward exactly as it did before. This is the payoff of compacting hard rather than often: one invalidation buys many cheap turns. If instead the system had compacted a little every turn, it would have invalidated the cache on every single call and paid a summarization model call each time, spending more than the tokens were worth. The rhythm you want is long stretches of cheap cached growth punctuated by infrequent, decisive compactions.