Design layered memory that preserves evidence boundaries

Pattern 08 companion: Python example · Behavioral tests · Example license

Remembering more can make an answer worse

An incident assistant sees a growing queue and retrieves a successful restart from last month's outage. The old incident involved a different worker version, and the current runbook says a restart would lose an uncommitted batch. A memory system that ranks by textual similarity alone can confidently recommend the wrong intervention. The problem is not merely retrieval accuracy; it is whether the retrieved item is applicable and authorized for this decision.

Our proposed design separates working notes, incident evidence and runbooks. Working notes help one investigation stay coherent. Incident records describe what happened in a particular context. Runbooks express maintained operational guidance. A retrieved incident is not a standing policy, and a generated working note is not an authoritative observation. Preserve these distinctions in storage and in the context supplied to any future model adapter.

Architecture and request sequence

Component diagram for Design layered memory that preserves evidence boundaries — Original reference architecture by Sandeep Belgavi. Arrows show the responsibilities and transitions described below.

Request sequence for Design layered memory that preserves evidence boundaries — Illustrative sequence for this design; failure and recovery variants are explained in the article.

Scope before similarity

The reference applies tenant, revision and expiry filters in the database before calling the scorer. This order matters. A powerful ranking model should never receive another tenant's record just to decide that it is irrelevant. Similarly, an expired or incompatible runbook should not enter the candidate set and rely on a later prompt to reject it. Authorization and applicability are hard boundaries; ranking is a preference inside those boundaries.

LangChain's memory overview distinguishes thread-scoped state from memory shared across sessions through namespaces. Our design adds explicit evidence fields and a strict revision filter for this incident example. Those choices are original application policy, not a claim that every memory framework provides the same tenant enforcement or revision semantics automatically.

Give each record an evidence contract

Every record contains a tenant, identifier, revision, kind, body, source and expiry. The source is a citation locator; revision indicates compatibility with the incident context; expiry defines when the record stops being eligible. The teaching store uses one row per tenant and identifier. Replacing a row updates its active representation, which is convenient for the example but is not an immutable audit history.

In a larger application, distinguish document revision from target-system revision. A runbook may be revision seven while applying to worker releases four through six. Our single revision field intentionally avoids that complexity. If you need ranges or compatibility rules, model them explicitly and test their boundaries. Do not overload a text tag with undocumented semantics and expect an embedding model to infer whether operational guidance is current.

Keep recall and authority separate

The original TokenScorer counts overlapping lowercase alphanumeric tokens. This is a deterministic lexical baseline, not semantic search, embeddings or an LLM. It is useful because the retrieval contract can be tested without a provider. A future adapter can replace scoring while preserving the rule that only eligible records reach it and every returned passage keeps a citation.

Even a highly relevant passage is still data. A log line saying to ignore policy must not become a system instruction; a past operator's workaround must not automatically grant permission for a new action. A response generator should cite retrieved evidence, identify uncertainty and use a separate authorization gateway for tools. A citation demonstrates where text came from, not that the text is true, safe, complete or applicable beyond the enforced filters.

Capacity and budget assumptions

For a small trial, imagine ten thousand evidence records per team and a maximum of three returned items per query. The reference scans eligible rows and scores them in Python, which is transparent but unsuitable as a blanket scaling recommendation. Measure candidate count and latency before adding an index. A vector database may improve retrieval operations while introducing indexing delays, access-control configuration and deletion propagation work.

Bound input size and context size separately. Limiting the number of returned records does not limit token count if one body is enormous. A production adapter should enforce byte or token limits, excerpt selection and stable citation spans. Also separate retention cost from inference cost: a cheap archive can still produce expensive model context if every query retrieves large passages. Choose budgets based on measured incident questions, not the maximum context window advertised by a model.

Define success with adversarial fixtures

The original store should return one eligible queue runbook while excluding an expired incident and another team's private note. Its tests also cover revision mismatch, deterministic ties, empty matches and scoped deletion. These are retrieval-contract checks, not evidence that the resulting incident diagnosis is correct. Use the evaluation layer to assess answer grounding separately.

Add fixtures where a highly similar record belongs to another tenant, where an older runbook has more matching words, and where retrieved text contains hostile instructions. The desired behavior is not always more recall. Sometimes the correct result is no eligible evidence and a request for fresh investigation. Design the caller to handle that empty result explicitly instead of filling the gap with stale memory or a fabricated citation.