Evidence label: deterministic synthetic model. This page runs a small JavaScript model of a long-running agent. Each synthetic task takes a random number of steps; each step adds a tool result of random size, and about half the steps reveal a fact that the final answer might need. When the context reaches the trigger, the chosen strategy drops old messages or summarises them under a token budget, optionally after clearing old tool results and with a pinned notes file. Every step is billed on the context it reads. Nothing here calls a model; all numbers are calculated estimates.
Play the lab
How to play
Set the window, the trigger and the strategy, then the summary budget, how many recent steps stay verbatim, and the two toggles for clearing old tool results and writing pinned notes. Every change replays the same tasks with the same seed, so differences come from your change. Tiles show task success, cost per task, the share of needed facts that were lost, how often the original instruction was lost, compactions per task and the p95 peak context. Use New random seed to check that a conclusion holds.
Missions unlock in order. Load mission setup starts from a broken configuration. Events triple the size of tool outputs, double task length, or make the summariser need three times as many tokens per fact.
What the model assumes
Tasks average 40 steps (80 with the long-horizon event). Tool results are lognormal around 1,500 tokens; each step adds a 200-token assistant message. The system prompt is 3,000 tokens and the instruction 500. Truncation drops the oldest messages, starting with the instruction, until the context is half the window. Summarisation condenses everything except the recent steps into the summary budget, keeping each fact with probability 1 − exp(−budget / (40 × facts)), re-applied to facts already in a summary, and keeps the instruction with probability 0.99. A cleared tool result keeps its fact only if the assistant restated it (60%). The agent notes 90% of facts when notes are on, at 60 tokens each. Final answers succeed only if every needed fact survived, with a reliability that falls from 97% to 82% as the final context grows to 200k. Input is billed at an illustrative $3 per million tokens.
What it cannot tell you
Real summarisers do not lose facts independently, and what they drop depends on the prompt and the content. Real agents do not know in advance which facts matter, and the model here ignores prompt caching, sub-agents, retrieval of old turns and the quality of the summary text. Use the lab to understand the shape of the trade-offs, then test your own policy with tasks whose answers depend on early facts.
Read the full explanation
- Agent Context Compaction: What to Keep When the Window Fills Up (companion article)
- Why Agent Loops Run Away and How Step Budgets Stop Them
- Agent Memory That Does Not Rot: Write Policies, Decay and Conflicts
All AI in Production labs · Lab catalog
- Lost in the Middle: How Language Models Use Long Contexts (TACL 2024)
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- MemGPT: Towards LLMs as Operating Systems
- Anthropic: Effective context engineering for AI agents
- LangGraph documentation: memory concepts
Views are my own and do not represent any employer, past or present.