Why architecture matters here
Three converging facts made context engineering a discipline. First, attention is not uniform: models attend most reliably to the start and end of context (the lost-in-the-middle effect), long contexts dilute focus across everything including noise, and one confidently-wrong retrieved document can poison an otherwise-correct reasoning chain. More context is not monotonically better; past a workload-dependent knee, quality declines while cost rises — the worst slope in engineering. Second, agents generate their own bloat: every tool call's result enters the context and, naively, stays for the conversation's remainder — a session that touches five APIs and reads three documents accumulates hundreds of kilobytes of payload the model re-reads every subsequent turn. Third, economics compound per turn: context tokens are billed on every call, so a 100k-token context on a 40-turn conversation costs 4M input tokens — and prompt caching only rescues the prefix that stays stable, which is itself a context-engineering property you design for or forfeit.
The failure modes are distinctive enough to have names. Context rot: quality degrading over a long session as stale tool dumps and dead-end explorations crowd the window. Context poisoning: an early hallucination or bad retrieval that later turns keep citing as established fact. Goal drift: the original instruction, buried mid-context under accumulated history, losing the position battle to whatever arrived last. Each is an architecture problem — what was allowed to persist, where — not a prompt-wording problem, and no instruction tweak fixes what the window's composition broke.
The reframe that makes the discipline tractable: the context window is a per-turn budget with an allocation policy, like memory in an embedded system. Identity and rules get a fixed reservation; tools pay rent per declaration (catalog size is a context decision); history gets a windowed slice with compaction beyond; retrieval gets a scoped, ranked allowance; and everything else stays outside behind references until explicitly pulled. Teams that write this policy down — and meter it — ship agents that stay sharp at turn 60; teams that don't, ship agents that were impressive at turn 5.
The architecture: every piece explained
Top row: the standing allocations. The system prompt holds identity, rules, and output conventions — kept stable (cache prefix!) and lean: policy that can be enforced in code (guardrail callbacks) shouldn't spend prompt tokens being begged for. Tool declarations are the surface the model routes on; every tool's schema and description bills every turn, which is why a 40-tool catalog is a context decision — filtered toolsets and sub-agent decomposition are context moves as much as architecture moves. The history window carries recent turns verbatim — the working conversation — sized in tokens, not turns. Retrieved context arrives just-in-time: RAG chunks, memory hits, file contents — scoped to the current question, ranked, capped, and ideally labeled with provenance so the model can weigh trust.
Middle row: the maintenance machinery. Compaction summarizes aged history: when the window's history allocation overflows, older spans are distilled — decisions, unresolved threads, key facts — into a summary block that replaces them; engineering the summarizer (what it must preserve: commitments, identifiers, user preferences) is a first-class prompt task with its own evals. Tool-result hygiene is the biggest single win in most agents: after_tool truncation, field extraction (the model needed 6 fields, not the 300-field payload), artifact offload with references, and aging-out of stale results (the search results from turn 3 need not survive to turn 30). Structured state moves load-bearing facts out of prose: order IDs, chosen options, and workflow position live as state keys injected via instruction templating — surviving compaction by construction, immune to summary lossiness. Sub-agent isolation is context firewalling: an exploration that will burn 80k tokens (read this codebase, scan these documents) runs in a child agent's fresh window and returns a 500-token conclusion; the parent's context receives the distillate, never the debris.
Bottom rows: the physics and the payoff. Attention effects shape ordering: instructions and current-task framing at the edges (start via system prompt, end via the latest message), bulk reference material in the middle where per-token attention is cheapest to sacrifice; critical constraints restated in the final position when they must win. Caching alignment turns composition into money: stable prefix first (system prompt, tools — byte-identical across turns), variable content last; a context assembled in that order caches its expensive majority, and one careless timestamp in the system prompt forfeits it. The ops strip closes the loop: per-turn token budgets by component (measured, alerted), context evals (does quality hold at turn 40? does compaction preserve the commitments?), and drift monitoring for the slow bloat that every living agent accretes.
End-to-end flow
Engineer a real agent's context and watch the budget work. A support agent: 200k-token model, but the team sets a working budget of 24k per turn — quality and cost both prefer it. Allocation: system prompt 1.2k (identity, rules, output format — stable for caching); tools 2.8k (nine tools, descriptions curated); history window 10k (verbatim recent turns); retrieval allowance 6k (KB articles + memory hits, top-3 ranked, provenance-labeled); state-templated facts 0.5k; headroom 3.5k for the model's reasoning-friendly margin.
Turn 12 of a warranty dispute: the user uploads a 40-page manual PDF. Naive design would inline it (60k tokens, budget dead). Instead: artifact storage, a 90-token reference in context, and an extraction tool the model calls with targeted questions — two calls later, 800 tokens of relevant clauses entered context; 59k never did. Turn 23: the history allocation overflows; compaction fires, replacing turns 1–14 with a 340-token summary explicitly preserving: dispute ID, purchase date, the two commitments made ('replacement approved if repair fails', 'callback by Friday'), and the user's stated deadline. An eval case exists for exactly this: replay the session, assert the post-compaction agent still honors both commitments — because a summarizer that drops commitments is a liability generator.
Turn 31 needs deep research: 'is this failure mode covered under the 2024 policy revision?' The orchestrator spawns a research sub-agent whose fresh window inhales the policy corpus (70k tokens), and returns a 400-token answer with citations; the parent's context never sees the corpus. Meanwhile the cache line holds: the 4k stable prefix (prompt + tools) is byte-identical across all 31 turns — 90% of it billed at cache rates; the one engineer who once added a Current time: ... line to the system prompt (invalidating the cache every call, +$1,100/week) is now the team's favorite cautionary tale. The dashboard that guards all of it shows per-turn tokens by component; the week retrieval quietly doubled (a KB re-chunking regression pushed chunk sizes up), the alert fired before the bill did, and the fix was a ranking config, not an incident. At turn 47 the conversation resolves — the agent as sharp as at turn 5, because the window at turn 47 contains almost nothing that isn't load-bearing.