Why architecture matters here
Without a memory tier, an agent has exactly two bad options. It can stuff the entire history of every past conversation into the prompt, which blows the context window, re-pays the token cost of the whole relationship on every single turn, and drowns the model in irrelevant detail so that the one fact that matters is buried under a thousand that do not. Or it can start every session from zero, which makes it feel amnesiac and forces the user to re-establish context they already gave. Neither scales. The memory tier exists to make recall selective: store much, but carry into context only the few fragments the current turn actually needs.
Three properties fall out of separating memory from the session. Recall is bounded on the hot path: the session stays inside the context window because long-term knowledge is fetched on demand, not permanently resident. Recall is semantic: memory is searched by meaning, so a user asking about 'the deployment we discussed' finds the relevant past exchange even though they never used the word the agent stored it under. And recall is governed: because memory is an explicit store with its own lifecycle, you can scope it per user, expire it, audit it, and delete it — none of which is possible when 'memory' is just an ever-growing transcript smeared through the prompt.
The trade-off you operate around is that memory introduces retrieval error into the loop. A session is ground truth — it is literally what was said. A memory search is a ranked guess: it can miss a relevant fragment (hurting continuity) or surface an irrelevant or stale one (polluting the prompt and, worse, letting the model act on outdated facts). The engineering job is to make that guess good enough to help and cheap enough to always run, while never letting it leak across the boundaries — user, application, tenant — that separate one person's remembered life from another's.
The architecture: every piece explained
Top row: the write path. Every conversation is a Session whose turns the SessionService appends and reloads. When a session is worth remembering — at its close, or periodically while it runs — ingestion selects the salient turns rather than everything: the user's stated preferences, decisions reached, facts established, not the small talk and the tool-call scaffolding. Those selected fragments are handed to the MemoryService via addSessionToMemory. Distillation is the quiet lever here: store raw turns and you fill memory with noise; summarize or extract structured facts and you store less, retrieve more precisely, but risk the summarizer dropping something that later mattered.
Middle row: how a fragment becomes searchable. Each stored fragment is run through an embedder that maps its text to a vector, and that vector lands in a vector store — an approximate-nearest-neighbour index that answers 'what is semantically closest to this query'. Alongside it, a metadata store keeps the structured facets: which user, which application, when it was written, what kind of memory it is. Those facets are not decoration — they are the filter that makes searchMemory safe, because a query is always scoped to the calling user and app before similarity ranking ever runs. The load_memory tool is the model-facing surface: ADK exposes recall as a tool the agent can choose to call, so retrieval happens when the model judges it needs history, not on every turn blindly.
Bottom rows: the read path. The retriever takes the query, applies the mandatory user/app filter, runs the similarity search, ranks the candidates, and deduplicates near-identical fragments so the same remembered fact does not appear five times. The surviving fragments go to the prompt assembler, which injects them into the model's context as clearly-delimited recalled memory — labelled as retrieved history, not as the user's current words, so the model treats it as background rather than instruction. The ops strip underneath lists the controls that keep the whole thing honest: hard scope isolation between users and tenants, retention and TTL so memory does not accumulate forever, relevance metrics that tell you whether recall is helping, and PII handling for a store that by definition holds what people told the agent about themselves.
One distinction the diagram compresses is worth drawing out: not all memory is the same shape. Some deployments store semantic memory — distilled facts and preferences, embedded for fuzzy recall, which is what the vector path above handles. Others also keep episodic memory — pointers back to whole past sessions, so the agent can say 'we did this on Tuesday' and reload the actual exchange. And structured facts — the user's timezone, their default region — may live in a plain key-value profile rather than a vector index at all, because you want them retrieved by exact key, not by similarity. A mature MemoryService is really a small router: it decides, per query, whether the answer is a looked-up profile field, a semantic search, or an episodic reload, and blends the results. Treating every memory as one undifferentiated vector blob is the commonest reason recall feels vague — a birthday should be a fact you know, not a fuzzy neighbour you hope ranks first.
End-to-end flow
Trace a returning user. Two weeks ago, in session A, a developer told the agent 'we deploy to the eu-west region and we never use us-east for latency reasons'. At the close of session A, ingestion judged that a durable preference, distilled it to a compact fact, embedded it, and stored it in the MemoryService scoped to that user and application, tagged with a timestamp and a 'preference' type. It has sat in the vector index ever since, carried in nobody's context, costing nothing per turn.
Today the same user opens session B and asks 'can you scaffold a new deployment config for me?'. The agent's session holds only this new conversation — it has no idea about the region preference in its immediate context. But the agent has the load_memory tool, and the task of writing a deployment config is exactly the kind of thing prior preferences bear on, so the model calls the tool with a query like 'deployment region preferences for this user'. The retriever scopes to this user and app, embeds the query, searches the vector store, and the two-week-old fragment ranks at the top. It comes back as recalled memory, the prompt assembler injects it as clearly-labelled history, and the agent scaffolds a config defaulting to eu-west and explicitly avoiding us-east — without the user re-stating any of it.
Now the write side of this very turn. As session B produces new durable facts — say the user names the new service and picks a runtime — ingestion can add those to memory too, either at session close or on a rolling basis, so the next conversation inherits them. Note the asymmetry: writing to memory is a background, best-effort concern that can be batched and retried, while reading from memory is on the latency path of a turn and must be fast and bounded. Systems that get this backwards — synchronously blocking a turn to write a summary, or lazily indexing so recent facts are not yet searchable — feel sluggish or forgetful in exactly the ways the memory tier was supposed to fix.
Consider the stress path. Suppose the user's preference has since changed — last week, in session C, they said 'actually we've moved everything to us-east now'. If both fragments live in memory, a naive search returns both, and the model may act on the stale one. This is why timestamps and recency-aware ranking matter, and why some deployments prefer to supersede rather than merely append — updating or invalidating the old preference when a contradicting one arrives. Handle it well and the agent recalls the current truth; handle it badly and long-term memory becomes a source of confident, outdated mistakes.
Watch the read side once more, because it is where the router idea earns its keep. When the model calls load_memory for 'deployment region preferences', a naive implementation embeds that string and runs a single vector search. A better one recognizes that a region default is exactly the kind of stable preference that may also live in the structured profile, checks there first for an exact answer, and only falls back to semantic search for anything the profile does not hold. The two results are merged and de-conflicted — if the profile says eu-west and a semantic hit agrees, confidence is high; if they disagree, recency and source authority break the tie. This blended retrieval is invisible to the agent, which simply receives a small, clean set of recalled facts, but it is the difference between an assistant that reliably knows the user and one that occasionally guesses right. The cost is a slightly more complex retriever; the payoff is recall that feels like memory rather than search — and it is why the ingestion side is worth the effort of classifying a fact as profile, semantic, or episodic at the moment it is first stored, rather than dumping everything into one index and hoping ranking sorts it out later.