Agent Memory Layers Compared

Working memory

The model’s context window. Token-limited. Includes system prompt, recent turns, retrieved context. Designed per-turn, not persisted.

This is the runtime activation of the agent’s mind — what the LLM sees and reasons over in a single forward pass. It consists of the system prompt (instructions, role, constraints), the conversation history from this session (sometimes summarized), any retrieved context from a vector database or knowledge store, and the current user input. The context window is both a blessing and a constraint: it gives the model enough information to reason coherently, but it is finite. A Claude Haiku model has 200K tokens; even larger models have hard ceilings.

Because working memory is designed per turn and never persisted, it is rebuilt fresh each time an agent takes a step. That rebuild is an expensive operation when the history is long, which is why most agent frameworks implement summarization: as the conversation grows, earlier turns are condensed into a summary (often by the LLM itself) so the context window does not bloat. The risk of summarization is information loss — a crucial detail from turn 3 might vanish when turn 50 is summarized — so the practice is to keep raw history up to a certain length, then summarize older turns, then eventually drop the summaries once the session is truly ancient.

Working memory is where every decision happens. It is also the most expensive layer to operate, because feeding the model text costs tokens and tokens cost money. Efficient agents minimize working memory size by keeping sessions short, summarizing aggressively, and being selective about what context is retrieved.

Structure of working memory. A typical agent’s working memory looks like: a system prompt (perhaps 1-2K tokens), the last 10-20 turns of conversation (the “raw window”), one or more summaries of older turns (each 500-1000 tokens), maybe a retrieval context from a semantic store (200-1000 tokens of relevant facts), and the current user message. The art is balancing all these pieces so that the model has enough information to reason well but not so much that you waste tokens and money on redundancy or noise.

Practical limits. At around 100K tokens of history, many teams start to see diminishing returns: the LLM has more context than it can meaningfully weigh, and adding more just inflates the bill. Agents that run for hundreds of turns typically implement an aggressive summarization strategy: keep the last 3-5 turns verbatim, summarize turns 5-20 into a single paragraph, summarize turns 20-50 into another, and throw away anything older unless it matters for the current task. The key is tracking what is important: if turn 8 resolved a hard constraint that the current task depends on, it stays; if it was a false start that the agent walked back, it can be dropped.

Episodic memory

Conversation history. Sessions. Often summarized as length grows. Stored in fast KV (Redis, DynamoDB). Visible to current session; ephemeral.

Episodic memory is the agent’s account of what happened in this conversation: every turn the user took, every tool call the agent made, every result that came back. Unlike working memory (which lives only in the current LLM context), episodic memory is persisted in an external store and can span multiple sessions or conversations.

The storage tier is typically a fast key-value database like Redis or DynamoDB, indexed by session ID or conversation ID. The data structure is simple: a list or log of events, often with metadata (timestamp, speaker, token count, cost). This layer answers the question: “What have we done together before?”

Because episodic memory is append-only and can grow unboundedly, most systems implement a retention policy. Old conversations are summarized or truncated, recent ones are kept intact, and the oldest are deleted after a threshold (days, weeks, or months depending on use case and compliance). Some agents implement a two-tier pattern: keep raw history for the last 10 conversations, summarized history for the 10 before that, then discard.

Episodic memory is ephemeral in the sense that it is session-bound: it does not carry semantic meaning across conversations. An agent might remember “In our chat on June 15 you asked about Kubernetes,” but it does not infer broader lessons or learn domain-specific patterns. That is the job of the next layer.

When to retrieve episodic memory. Episodic memory becomes useful when a user returns and wants to reference prior conversations. An agent system might load the last few conversations for a returning user and inject a summary into working memory (“In June you were working on scaling a PostgreSQL cluster; we settled on read replicas and connection pooling”) so the LLM has the context to ask smart follow-ups or avoid rehashing old ground. This is cheaper than having the user explain the full history again and faster than the agent re-discovering the constraints.

Compliance and retention. Episodic memory often carries user data, so it falls under GDPR, CCPA, and other privacy regulations. A retention policy is not just an engineering choice; it is a legal requirement in many jurisdictions. Teams typically store raw conversation for 30-90 days, summaries for up to a year, then delete everything unless a user requests it be archived. Compliance logging (audit trails of what the agent did) is separate and often kept longer.

Semantic memory

Long-term facts about the user, world, past resolved issues. Vector DB for retrieval + structured DB for facts. Updated explicitly (or with user confirmation). Survives across sessions, devices, devices.

Semantic memory is the agent’s knowledge base: facts, patterns, and inferences that persist across conversations and that the agent applies to new problems. This is where an agent learns. If working memory is the current thought, and episodic memory is the record of what happened, semantic memory is the distilled wisdom.

Semantic memory is typically dual-storage: a vector database (like Pinecone, Weaviate, or Milvus) for retrieval-by-similarity, and a structured database (PostgreSQL, MongoDB, or a triple store) for facts. A typical entry might be indexed both ways: “User prefers API-first architecture” is stored as a vector embedding (so it surfaces when discussing architecture choices) and as a structured fact (so the agent can reason over it logically and even contradict it if evidence warrants).

Crucially, semantic memory is updated explicitly. It does not grow automatically by inferring from episodic memory. Instead, at the end of a conversation or after resolving a problem, an agent (or a human curator) decides: “What did we learn that is worth keeping?” The update might be a new fact about the user (“Prefers TypeScript over Python”), a resolved issue (“CORS errors in Next.js: solution is to enable credentials in fetch options”), or a domain pattern (“This customer always needs geo-distributed failover”). Some systems ask the user to confirm updates before they stick; others have the LLM propose updates and a human approves them before committing to the persistent store.

Because semantic memory is global and shared, it scales across users and time. A fact learned from one user helps the agent serve the next. Over months or years, the semantic store becomes the agent’s true intelligence — far more valuable than any single conversation.

Building semantic memory deliberately. The best-performing agents invest in semantic memory early and treat it as a first-class product, not an afterthought. Every resolved issue should generate a fact: not just “CORS error fixed,” but “CORS error in Next.js with SameSite cookies: add credentials: 'include' to fetch; set Access-Control-Allow-Credentials: true on the backend; ensure origin is not 'null'.” Every discovered user preference gets indexed: “Prefers concise explanations without jargon” or “Wants all SQL queries explained with query plans.” The more specificity, the better the retrieval.

Cost-benefit of semantic memory. Semantic memory compounds. The first conversation is expensive: the agent explores, asks clarifying questions, learns slowly. By conversation ten, if the semantic store is rich, the agent can retrieve relevant facts immediately and spend tokens on new problems instead of re-learning the user’s preferences. A single fact well-placed might save 500 tokens across the next 20 conversations, easily 10,000 tokens over a year. That is millions in savings for large agent deployments. The payoff is asymmetric: early investment in curation and tooling pays off exponentially.