Why architecture matters here
Token count drives the three costs that matter. Money: input tokens are the bulk of spend in most agentic and RAG workloads — a 40k-token prompt answered in 300 tokens pays for the prompt, not the answer, on every single call. Latency: prefill time grows with input length, so a bloated prompt directly slows time-to-first-token — often the difference between an interactive product and a sluggish one. Quality: the lost-in-the-middle effect is real and measured — salient facts buried mid-context get used less reliably than the same facts in a short prompt. Compression, done right, improves all three at once, which is why mature LLM platforms treat it as core serving infrastructure rather than an optimization to get to later.
Done wrong, it is a silent quality killer, and the failure is always the same shape: something load-bearing was compressed. A truncation cut off the output-format instructions, so JSON parsing started failing at 2%. A summarizer paraphrased a safety rule into vagueness. An aggressive retriever filter dropped the one document with the actual answer. These failures do not throw errors; they show up as a slow drift in downstream metrics weeks later, unattributed. The architecture exists precisely to make compression selective and verifiable: per-class policies say what may be touched, validators confirm invariants survived, and the eval harness measures answer quality at each compression level so aggressiveness is a dial with known cost.
There is also a systems-level interaction that makes naive compression actively counterproductive: prompt caching. Providers charge a fraction of the price for prompt-prefix tokens that were cached from a previous call, and cache hits require byte-identical prefixes. A compressor that rewrites the prompt differently on every call destroys prefix stability and can raise the effective bill while lowering token counts. Any real compression architecture must be cache-aware from the first design meeting — stable content first, volatile content last, recompression only at deliberate boundaries.
The architecture: every piece explained
The segment inventory parses the outgoing request into typed segments: system instructions, tool definitions, few-shot examples, conversation history (per turn), retrieved documents (per chunk), and the current user query. Each carries metadata — class, token count, age, and a mutability flag. The token budgeter owns the target: given a total budget (from config, model limits, or cost tier), it allocates per class — say, instructions untouched at 1,500, tools 2,000, history 4,000, documents 6,000. Budgets are config, not code, so cost tiers and A/B arms are a config diff. The relevance scorer supplies the ranking signal within a class: embedding similarity between each history turn or document chunk and the current query, so the compressor knows which tokens are expendable, not just how many.
Compression itself is an escalating cascade, cheapest first. The extractive pass drops whole units — the retrieved chunk scored 0.31 against the query, the tool the router says this request cannot need, the history turn from forty exchanges ago — achieving big reductions with zero rewriting risk. If the budget is still exceeded, the abstractive pass summarizes: a small, fast model condenses old conversation turns or verbose documents into dense summaries, preserving stated facts, decisions, and constraints per a class-specific summarization prompt. Only if still over budget does the token pruner run — LLMLingua-style perplexity-guided deletion of individual low-information tokens, powerful but the most likely to mangle structure, and therefore restricted by policy to prose segments only, never code, JSON, or instructions.
Two components handle time and money. The rolling memory makes long sessions sustainable: instead of re-summarizing the whole history every turn (quadratic cost, and drift with every rewrite), it maintains a hierarchical summary — recent turns verbatim, older turns in per-episode summaries, ancient history in one session-level digest — folding turns upward only when they age across a boundary. The cache-aware assembler then lays out the final prompt for prefix stability: system instructions and tool definitions (byte-identical across calls) first, stable summaries next, volatile retrieved chunks and the current query last. The validator makes the final check: instruction segments byte-identical to their originals, all JSON and code blocks still parse, required keywords present, budget actually met. Fail any check and the pipeline falls back to a less aggressive level — never silently ships a broken prompt.
End-to-end flow
Turn forty-one of a long support-agent session arrives: the user asks a follow-up about an error code mentioned much earlier. Uncompressed, this request would be 38,000 tokens — 1,500 of instructions, 3,000 of tool schemas, 26,000 of history, 7,000 of freshly retrieved documentation, plus the query. The budgeter's target for this cost tier is 12,000. The inventory tags every segment; the relevance scorer embeds the new query and scores all forty history turns and nine retrieved chunks against it. Three old turns — where the error code was first discussed — score high; most of the middle of the conversation scores near zero.
The cascade runs. Extractive: five of nine retrieved chunks fall below the relevance floor and are dropped whole; the tool router confirms this turn cannot need the account-management tools, so their schemas are omitted; history turns 5 through 30, already folded into the rolling memory's episode summaries on previous turns, are represented by those summaries rather than verbatim text — but the three high-relevance turns are pinned verbatim despite their age, because relevance overrides recency. That alone reaches 14,500 tokens. Abstractive: the two longest surviving documentation chunks are summarized by the small model with the "preserve error codes, commands, and version numbers exactly" document policy, landing the prompt at 11,800. The token pruner is not needed and does not run — the cascade stops at the first sufficient level.
The assembler lays out the result: instructions and the trimmed tool set first — byte-identical to the previous turn's prefix, so the provider's prompt cache hits through the first 6,000 tokens at a tenth of the price — then the stable episode summaries, then the pinned verbatim turns, retrieved summaries, and the query. The validator confirms the instruction hash matches, both remaining tool schemas parse, and the error code string itself survived compression (it is on the required-keywords list, extracted from the query). The request ships: 11,800 tokens instead of 38,000, first token arrives in a third of the time, and the answer cites the error code correctly. Asynchronously, one in fifty such requests is shadow-sent uncompressed too, and both answers go to the eval harness for judgment.
The whole traversal added a few tens of milliseconds of pipeline time — embedding lookups are cached, the small summarizer only ran on two chunks — against a prefill saving measured in seconds. That asymmetry is typical, but it must be measured per workload: a pipeline whose own latency approaches the prefill time it saves has inverted its purpose, and the budgeter should be told to stop earlier in the cascade.