Why architecture matters here
Caching matters because agent economics are dominated by repeated work, and caching is the primary lever against it. The system prompt and tool declarations are sent every single turn — identical bytes, re-billed each time — and for a long conversation or high-volume agent, that repeated prefix is a large fraction of the token cost. Prompt caching (the provider bills the cached prefix at a fraction of full price) can cut costs substantially with zero correctness risk (the prefix is byte-identical, so the cache is exact) — making it the first caching move for any agent. FAQ-shaped workloads (many users asking similar questions) have another repetition: the same answer computed repeatedly, which semantic caching can absorb. And expensive tool calls (a slow API, a heavy database query) called repeatedly with the same arguments are dedupe candidates. Caching is where agent cost optimization has the most leverage, because agents do so much repeated work.
The prompt-caching case deserves special attention because it's high-leverage, low-risk, and dependent on architecture you control. Providers cache the stable prefix of a request and bill it cheaply on subsequent requests that share it — but only if the prefix is byte-identical across requests. This makes prompt caching a context-engineering property: assemble the context with the stable content first (system prompt, tools — unchanging) and the variable content last (the conversation), and the prefix caches; put anything volatile in the prefix (a timestamp, a request ID, reordered tools) and the cache misses every time, forfeiting the savings silently. The classic incident is someone adding Current time: ... to the system prompt, invalidating the cache on every request and multiplying costs — a one-line change with a large bill. Cache-aligned context assembly (stable prefix, no volatile bytes above the variable line) is the discipline that makes prompt caching work, and it's entirely in the application's control.
And the correctness risk scales with the cache's cleverness. Prompt caching is safe (exact prefix match). Tool caching is mostly safe (exact key match on tool + arguments) but risks staleness (a cached lookup that's now outdated). Semantic caching is the riskiest: it returns a cached answer for a similar query, and 'similar enough' is a judgment — a query that's semantically close but materially different (asking about a different date, a different account, a subtly different question) getting a cached answer that's wrong for it. Semantic caching's power (absorbing FAQ variation) is also its danger (returning wrong answers for genuinely different queries), so it needs careful similarity thresholds, scoping (never leak one user's answer to another), and monitoring — the correctness discipline that separates a semantic cache that saves money from one that generates wrong answers at scale.
The architecture: every piece explained
Top row: the caching layers. Prompt caching reuses the stable context prefix: the model provider caches the prefix (system prompt, tool declarations) and bills it cheaply on subsequent requests sharing it — requiring cache-aligned assembly (stable prefix first, byte-identical across requests). Semantic caching returns a stored answer for a similar query: the query is embedded, compared to cached query embeddings, and if a sufficiently similar one exists, its cached answer is returned without a model call — powerful for FAQ workloads, risky on similarity judgment. Tool result caching dedupes tool calls: an expensive tool's result is cached by (tool, arguments), and a repeat call with the same arguments returns the cached result — saving the expensive backend call. Cache in callbacks: these live in ADK callbacks — before_model (check the prompt/semantic cache, short-circuit with a cached response) and before_tool (check the tool cache, short-circuit with a cached result) — the natural interception points.
Middle row: the mechanics. Cache keys define hits: exact (tool + arguments hash for tool caching; byte-identical prefix for prompt caching) or similarity-based (embedding distance threshold for semantic caching) — the key determines what counts as a hit and thus the savings-vs-correctness tradeoff. Invalidation and TTL manage freshness: cached entries expire (TTL) or are invalidated on data change, trading freshness (short TTL, fewer stale hits) against savings (long TTL, more hits) — the core cache tradeoff. Hit rate is the value metric: what fraction of requests are served from cache — the measure of whether the cache is worth its complexity and risk. Correctness risk: stale hits (outdated cached data) and wrong hits (semantic cache returning a cached answer for a materially different query) — the danger that scales with cache cleverness.
Bottom rows: scoping and hazards. Scoping is critical for correctness and privacy: per-user caches (a user's cached answers only for them) versus shared caches (answers shared across users) — a shared semantic cache leaking one user's personalized answer to another is a correctness and privacy breach, so scoping must match the data (shared only for genuinely user-independent answers, per-user for anything personalized). Cache stampede: when a popular cache entry expires, many concurrent requests miss simultaneously and all compute the expensive result at once (a thundering herd) — mitigated by locking (one request computes, others wait) or probabilistic early refresh. The ops strip: hit-rate monitoring (the value metric, tracked per cache layer), staleness management (TTL and invalidation tuned to freshness needs, stale-hit monitoring), and cost attribution (measuring the cost savings the cache delivers, justifying its complexity).
End-to-end flow
Layer caching into an agent and watch each layer's tradeoff. First, prompt caching (the easy win): the team ensures cache-aligned context assembly — system prompt and tools first (byte-identical every turn), conversation last — so the provider caches the ~4k-token prefix and bills it cheaply. On a high-volume agent, this cuts token costs substantially with zero correctness risk (exact prefix match). The one hazard they avoid: a developer nearly added a timestamp to the system prompt (which would have invalidated the cache every request, multiplying costs); code review caught it, and the rule became 'no volatile content in the cached prefix'. Prompt caching is the first move: high leverage, no risk, entirely in their control.
Semantic caching shows the savings-vs-correctness tension. The agent handles many similar FAQ questions, so the team adds semantic caching: embed the query, and if a very similar cached query exists, return its answer. Tuned with a high similarity threshold (only very close matches hit), it absorbs FAQ variation (many phrasings of 'what's your return policy' hit the cached answer) — real savings. But they hit the correctness risk: a lower threshold would catch more variation but also return cached answers for materially different queries (a query about a specific order's return status matching the generic return-policy answer — wrong). The tuning is conservative (high threshold, few but safe hits), and scoping matters: personalized answers (about a specific user's account) are per-user cached or not cached (never shared — one user's account answer must never hit for another), while generic answers (policies) are shared. The semantic cache saves money on generic FAQ while the scoping and threshold prevent it from leaking or mis-answering.
The tool-caching and operational vignettes complete it. An expensive tool (a slow third-party enrichment API, billed per call) is called repeatedly with the same arguments across users; tool caching (keyed by tool + arguments, with a TTL matching the data's freshness) dedupes these — the same enrichment result served from cache for the TTL window, cutting the expensive API calls dramatically. The TTL is tuned to the data's change rate (enrichment data is stable, so a longer TTL; volatile data would need shorter). A cache stampede is handled: when a popular tool-cache entry expires, locking ensures one request recomputes while others wait (rather than all hammering the API simultaneously). And the team monitors: hit rate per cache layer (prompt cache hit rate, semantic cache hit rate, tool cache hit rate — the value metric), staleness (stale-hit incidents, TTL tuning), and cost attribution (the savings each layer delivers). The consolidated discipline: prompt caching first (high-leverage, zero-risk, cache-aligned assembly), tool caching for expensive repeated calls (exact keys, TTL to freshness), semantic caching carefully (high similarity threshold, correct scoping, monitoring), and hit-rate and staleness monitoring throughout — because caching is the primary lever against agent cost, but its savings must be weighed against the correctness risk that scales with the cache's cleverness.