Agent Context Compaction: What to Keep When the Window Fills Up

A long-running agent accumulates context the way a log file accumulates lines. Every tool result, every intermediate thought and every retry is appended, and every step re-sends all of it to the model. Sooner or later one of three things happens: the context hits the window limit and the request fails, the bill per step climbs until the task is no longer worth running, or the model starts missing facts buried in a hundred thousand tokens of stale tool output. Compaction is the set of techniques that decide what stays in the window when it fills up.

This guide works through the arithmetic of context growth and why cost is the area under the context curve, compares the main strategies (truncation, clearing old tool results, summarisation, pinned notes and sub-agents), shows how fact loss compounds across repeated summaries, and lists the failure modes that only appear in production: dropped instructions, broken tool-call pairs, cache misses and summaries of summaries. The embedded lab runs synthetic long tasks through a fixed window so you can watch each choice trade cost against the facts the final answer needs.

Cost is the area under the context curve

In a standard tool-calling loop, step n sends the system prompt, the task and everything that happened in steps 1 to n − 1. If the fixed part is S tokens and each step adds g tokens (a tool result plus the assistant's message), the input at step n is S + g(n − 1), and a task of N steps reads about N·S + g·N²/2 input tokens in total. Context grows linearly; cumulative cost grows quadratically.

# illustrative values close to the lab defaults
S = 3,500 tokens (system prompt + task)     g = 1,700 tokens per step

N = 40 steps, no compaction:  40 * 3,500 + 1,700 * 40^2 / 2 =  1.50 M input tokens
N = 80 steps, no compaction:  80 * 3,500 + 1,700 * 80^2 / 2 =  5.72 M input tokens

# compaction keeps context between a floor F and a trigger T, so each step costs about (F + T) / 2
N = 80, F = 20k, T = 61k:     80 * 40.5k                     =  3.24 M input tokens
plus about 2-3 compaction calls, each reading ~61k and writing the summary

at an illustrative $3 per million input tokens: $17.2 versus about $10 per task

Doubling the task length quadruples the uncompacted cost. Compaction converts the quadratic into something close to linear, because the context oscillates in a sawtooth between the post-compaction floor and the trigger. That sawtooth is exactly what the lab draws. Prompt caching lowers the price of a stable prefix, which helps, but compaction rewrites the prefix, so the first step after each compaction pays full price again. Check your provider's caching rules when you model this.

Bigger windows do not remove the problem

The tempting fix is a model with a larger window. It postpones the overflow but makes two other problems worse. Cost per step keeps rising for as long as the context is allowed to grow, so a 200k window filled to 190k costs several times more per step than a 64k window that compacts. And models do not use long contexts uniformly. The "Lost in the Middle" study found that accuracy on multi-document question answering dropped when the relevant passage sat in the middle of a long input, and the RULER benchmark found that many models' effective context, the length at which they still perform well on retrieval and tracing tasks, is well below the advertised window. Results vary by model and change with each release, but the direction is consistent: a fact present somewhere in a huge context is not the same as a fact the model will use.

The lab captures this with a simple assumption, labelled as such: the chance that the final answer uses the context correctly falls linearly from 97% at an empty context to 82% at 200k tokens. Mission two shows the consequence. The team that bought the 200k window pays the most per task and does not get the best success rate.

The strategies and what each one loses

Every compaction strategy throws something away. The useful question is what it throws away and whether the task needed it:

Strategy	What stays	What is lost	Extra cost
Truncate oldest (sliding window)	The most recent turns	Everything early, including the task statement unless pinned	None
Clear old tool results	All assistant messages; stubs for old results	Details of old results that the assistant never restated	Re-fetching when a cleared result is needed
Summarise older turns	A summary plus the last k turns verbatim	Whatever the summary has no room for; compounds across compactions	One model call per compaction
Pinned notes (memory file)	Facts the agent chose to write down	Facts it did not recognise as important	Tokens per note on every step; a write call
Sub-agents	A condensed result from each sub-task	The sub-agent's full trajectory	Extra model calls; coordination overhead
Retrieval of old turns	An index of the full history	Anything retrieval does not surface	A search per lookup

Strategies combine. A common production stack pins the system prompt and task statement, clears tool results older than a few steps, lets the agent write important facts to a notes file that is always in context, and summarises the remaining conversation when a trigger fires. Anthropic's guidance on context engineering describes compaction, structured note-taking and sub-agent architectures in these terms, and some APIs and agent frameworks now ship tool-result clearing and memory tools as built-in features; check the current documentation of your stack, since these features are new and their parameters change.

How fact loss compounds across compactions

A summary has a token budget. If the region being summarised contains n facts the task might need, and preserving a fact takes on the order of k tokens, a budget B keeps each fact with a probability that saturates as the budget grows. The lab uses r = 1 − exp(−B / (k · n)) with k = 40 tokens: a 2,000-token summary over 20 facts keeps each with probability about 0.92. That is an assumption chosen to have the right shape, not a measured property of any summariser.

The important effect is compounding. Each compaction summarises the previous summary along with the new turns, so a fact from step 3 of an 80-step task passes through every compaction that follows. With four compactions and r = 0.92, it survives with probability 0.92^4, about 0.72. If the final answer needs four such facts, all four survive with probability about 0.27. That is how a summary that looks fine each time can still lose the task: no single compaction is bad, but the chain is.

Three things break the chain. A larger budget raises r for every fact, at the cost of a larger floor on every later step. Fewer compactions (a larger window or a higher trigger) reduce the number of multiplications, at the cost of more tokens per step. And taking the facts out of the chain entirely, by writing them to pinned notes as they are discovered, means no summariser ever gets the chance to drop them. Mission three is built around this: a weaker summariser that needs three times more tokens per fact, and two ways to fix it, one of which is much cheaper.

A compaction policy in code

A policy needs a trigger, a protected set, a summarisation step and a guarantee that the result is a valid conversation. The sketch below shows the shape; the token counter, message format and summariser prompt are yours to fill in:

TRIGGER = 0.80        # fraction of the window; leave room for the compaction call itself
KEEP_RECENT = 4       # turns kept verbatim
SUMMARY_BUDGET = 2000 # max tokens for the summary

def maybe_compact(state, window):
    if count_tokens(state.messages) < TRIGGER * window:
        return state
    pinned = [state.system, state.task]                # never summarised, never dropped
    head, tail = split_keep_recent(state.messages, KEEP_RECENT)
    tail = start_at_turn_boundary(tail)                # never split a tool call from its result
    summary = llm(
        "Summarise the work so far for yourself. Keep every identifier, number, "
        "decision and open question. Drop raw tool output.",
        state.previous_summary, head, max_tokens=SUMMARY_BUDGET)
    state.previous_summary = summary
    state.messages = pinned + [note_block(state.notes), summary_block(summary)] + tail
    log_compaction(before=count_tokens(head), after=count_tokens(summary))
    return state

def clear_old_tool_results(messages, keep=4):
    for m in tool_results(messages)[:-keep]:
        m.content = f"[result of {m.tool_name} cleared; call again if needed]"

Two details in that sketch prevent common incidents. Splitting at a turn boundary matters because most chat APIs reject a tool result whose matching tool call was removed, so naive truncation can turn a context overflow into a hard API error. And the trigger must leave headroom: the compaction call itself has to read the context it is compacting and write a summary, so a trigger at 98% of the window can fail exactly when it is needed.

Choosing the trigger, the budget and what to keep verbatim

The three numbers interact. A low trigger keeps every step cheap and the context well inside the region where models use it reliably, but it compacts often, and each compaction costs a model call and multiplies the fact-loss chain. A high trigger compacts rarely but makes every step expensive and spends more time in long contexts. The summary budget sets the floor of the sawtooth: a 6,000-token summary costs 6,000 tokens on every subsequent step, so making it larger than needed is not free.

The number of recent turns kept verbatim protects the agent's working memory: the last tool result it is reasoning about, the file it just opened. Too few and it re-calls tools to re-read what was cleared; too many and the floor rises. In the lab, clearing tool results with only one or two steps kept makes the agent re-fetch often, and each re-fetch is an entire extra step.

A practical starting point is to set the trigger between 60% and 80% of the window, size the summary from the number of facts a typical task accumulates (count identifiers and decisions in real trajectories), keep three to six recent turns, and then tune against two numbers: cost per task and success on tasks whose answers depend on early facts.

A worked example: Belgavi's Book Store catalog agent

Suppose a hypothetical agent for Belgavi's Book Store reconciles supplier feeds with the catalog. A typical task takes 80 steps; each step reads a supplier page or a catalog record of about 1,500 tokens and adds a 200-token note. The final step writes a report that must cite four specific ISBNs and prices found along the way.

With a 200k window and no compaction until 95%, the context grows to about 140k by the end; the calculation above gives roughly 5.7 million input tokens, about $17 per task at the illustrative price, before any output. Switching to a 64k window with a summary budget large enough to hold the facts keeps each step near 40k on average: about $10 per task. If the summariser is weak with numbers, the four ISBNs are exactly what it drops, and the report is wrong even though nothing errored. The cheap fix is to have the agent write each ISBN and price to a pinned notes block when it finds them, about 60 tokens each, rather than to enlarge every summary. Adding tool-result clearing (keeping the last two to four results verbatim) cuts the context again, because supplier pages are useful only until the relevant line has been noted.

Every figure here is calculated from the illustrative model; your own token counts and prices will differ. The decision sequence carries over: pin the task, take important facts out of the summary chain, clear bulky output after it has been used, and size the window by cost rather than by the largest available.

Failure modes and how to detect them

Compaction failures rarely raise errors; they show up as quietly worse answers. The ones worth instrumenting:

Dropped instruction. A sliding window removes the first user message. The agent keeps working, on a task it no longer remembers. Detect by checking that the task statement is present after every compaction, and pin it.
Summary drift. Summaries of summaries lose specifics first: numbers, identifiers, negative results ("supplier X has no stock"). Detect with evaluation tasks whose answers depend on facts from early steps, run at several task lengths.
Re-work loops. After clearing or compaction, the agent calls the same tools again to recover what it lost. Track repeated tool calls per task; a jump after a compaction change is a signal.
Broken conversations. Truncation that splits a tool call from its result produces API validation errors. Count them by compaction strategy.
Cache misses. Every compaction changes the prefix. If cached-token share drops sharply, compactions may be too frequent.
Compaction failing at the limit. A trigger too close to the window makes the compaction call itself overflow. Alert on compaction errors, not only on task errors.

The core dashboard is small: context tokens by step (the sawtooth), compactions per task, cost per task, success on long tasks, and the repeated-tool-call rate.

What the lab missions ask you to do

Mission one starts with a naive buffer that drops the oldest messages, including the user's request, and asks you to fix the task without a larger window. Mission two turns on long-horizon tasks with a 200k window and asks you to cut cost while keeping success: the answer is a smaller window and a better summary, not a bigger one. Mission three makes the summariser lose details and asks for a cheap way to keep the facts. Mission four floods the context with huge tool outputs and asks for a success target on a tight budget, which requires clearing old results without clearing so aggressively that the agent keeps re-fetching them.

Run the lab: Context compaction: keep the facts, drop the bulk

The lab runs a few hundred synthetic agent tasks through a context window. The chart traces the context size of one sample task step by step: the red line is the window, the amber line the compaction trigger, and purple dashed lines mark compactions. Facts appear in tool results as the task runs, and the final answer needs the original instruction plus four of them. Start with the naive buffer, then load mission one.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

Log context tokens per step for a sample of real tasks and plot the curve; compute cost per task as the area under it.
Pin the system prompt and the task statement so no compaction or truncation can remove them.
Set the compaction trigger with headroom for the compaction call itself, between 60% and 80% of the window as a starting point.
Add evaluation tasks whose answers depend on facts from early steps, and run them at several task lengths after every compaction change.
Give the agent a pinned notes block or memory file for identifiers, numbers and decisions, and clear old tool results once noted.
Make truncation turn-aware so a tool result is never separated from its tool call.
Track compactions per task, repeated tool calls and cached-token share alongside cost per task.

Sources and further reading

Related in this series:

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.