Multi-turn Jailbreaks and Memory Poisoning: Preserve Provenance Across Time

A conversation is not a permission record

A user can establish a harmless context over several turns and later request an action that exceeds it. The assistant may infer continuity: the person who discussed an account must be allowed to change it, or the person who described an emergency must be authorized to bypass a check. Those inferences are conversationally plausible and operationally dangerous.

Persistent memory makes the problem last longer. If a model writes “this user is an administrator” after reading an unverified claim, future sessions may receive it as an established fact. A secure application distinguishes remembered preferences, asserted facts, verified attributes, and current entitlements. Only an authoritative identity service determines the last category.

Define memory classes and writers

Avoid a generic key-value store where every tool can update every field. Put authority-bearing state in a separate service or namespace with independent access control. Naming a key “trusted” does not protect it if the same agent can overwrite it. Include the writer identity, evidence reference, creation time, and expiry in the record.

Some facts need correction rather than simple expiry. A payment can be reversed or an employee's role revoked. Design invalidation around the authoritative source's version or event stream. A stale but once-correct memory should not continue granting access after the underlying entitlement changes.

Compression can launder an instruction

Suppose a document contains a quoted instruction that the assistant is supposed to analyze. A summarizer may shorten it into a declarative rule, dropping quotation marks and source attribution. The next model sees a concise internal summary rather than the original external text. The text has become more influential without becoming more trustworthy.

Require summaries to preserve the distinction between claims and instructions. Keep provenance in structured metadata and attach original evidence identifiers. For high-impact workflows, reconstruct policy and identity from authoritative services at execution time instead of trusting a conversation summary. A shorter context can improve cost and latency while still making the security boundary worse if it erases these distinctions.

Make state transitions explicit

Illustrative python; adapt the persistence and identity adapters to your service.

ALLOWED = {
    "draft": {"evidence_collected"},
    "evidence_collected": {"review_required", "approved"},
    "review_required": {"approved", "rejected"},
    "approved": {"executed", "expired"},
}
def transition(old, new, trusted_actor):
    if new not in ALLOWED.get(old, set()):
        raise ValueError("invalid transition")
    if new == "approved" and trusted_actor != "policy_service":
        raise PermissionError("approval requires policy service")
    return new

This small function demonstrates transition restrictions, not a production state store. Persist transitions atomically, authenticate the actor outside the model, and reject stale writes using a version or compare-and-swap token. Otherwise two workers can both observe an old state and overwrite each other's decisions.

Keep the transition event separate from the conversational response. The assistant saying “approved” should not change the case state. The interface should display the backend's recorded status, including pending review or uncertainty, rather than infer it from generated prose.

Evaluate sequences, not just individual prompts

Construct synthetic conversations with setup, evidence, a legitimate action, a context summary, and a later privilege-changing request. Vary the number of turns and where the summary occurs. Include a session restart so the test exercises persisted memory rather than only the in-memory conversation. Record the full state before and after each step.

Use clean controls that legitimately update preferences or correct facts. An assistant must be able to learn that a user prefers another language without learning that the user can waive payment controls. The test suite should distinguish these operations and prove that allowed memory updates still work.

Prevent cross-session contamination

Scope memory by tenant, principal, purpose, and data classification. A global cache of personalized answers can leak facts even when the underlying database is correctly isolated. Cache keys need to include the access context that affected the result, and changes in authorization should invalidate affected entries.

Do not reuse a conversation identifier as an authorization token. Bind session access to the authenticated caller on every request. When an account is deleted or an employee leaves, account for long-lived summaries and derived memories as well as the primary session table. A memory service is a data product with lifecycle obligations, not an unstructured convenience layer.

Worked state migration: a preference becomes a false entitlement

A customer tells the assistant, “For future orders, remember that I handle our team's purchases.” Storing a preference such as preferred receipt language may be reasonable. Storing purchasing_administrator=true would be an unjustified promotion from conversational claim to entitlement. The difference is semantic, so the memory writer needs a schema that cannot express protected permissions in the first place.

Use separate storage interfaces: save_preference accepts a small set of user-managed fields, save_case_claim stores attributed statements, and the authorization service alone manages roles. The assistant cannot call an unrestricted update_memory operation on a shared object containing all three. At read time, the prompt may include the user's purchasing claim as context, but action authorization still queries current organizational membership.

Now simulate a context compaction. The original sentence is shortened into “User handles team purchases.” Preserve an attribution field stating that this is a user claim, not an identity-service fact. A later summarizer should not remove that distinction. Test a second compaction and a session restart, because provenance often disappears gradually rather than in the first transformation.

The user later asks to refund another employee's order. The assistant can ask clarifying questions or explain the required process, but the tool service checks the authenticated principal and order scope. The memory claim cannot broaden access. If the user actually receives a new organizational role, the authoritative service records it and a new policy decision can allow the action without rewriting conversational history.

For a schema migration, version memory records and explicitly map old fields. An old free-text note that looks like an entitlement should not be converted automatically into a new boolean permission. Route uncertain records for review or retain them as unverified claims. Test migration against adversarial notes and ordinary preferences.

The acceptance evidence includes attempts to write protected keys, cross-session reads, stale-role decisions, and deletion of derived memory. A passing result means conversational continuity improves the experience while authority remains current and independently verifiable. It does not mean the model will never repeat an incorrect remembered claim; output correction and permission enforcement remain separate concerns.

Contain and repair poisoned memory

Quarantine suspect records by writer or source lineage, then identify every derived memory that depends on them. Preserve a restricted incident snapshot before deletion where appropriate. Recompute safe summaries from verified records and invalidate caches that included the poisoned state. Ask users to reconfirm ordinary preferences only when the evidence actually requires it.

Measure unauthorized state transitions, stale-authority decisions, and cross-session leakage separately from response quality. The useful security property is that remembered conversation cannot manufacture present-day permission. Long-context research motivates broader testing, but the memory design and state machine here are original engineering controls for that application-level risk.

Practice and continue

Open the interactive companion lab to inspect the assumptions and outcomes with synthetic data. Then return to the 25-chapter AFP learning path.

• Indirect Prompt Injection in RAG: When Retrieved Data Becomes an Instruction
• Multimodal Prompt Injection: PDFs, OCR, Images, and Voice Boundaries

Sources and implementation scope

Primary references checked October 15, 2026. Framework details follow the linked documentation; the Northstar cases, policy contracts, and test designs are original illustrative designs. Python snippets labeled illustrative are not complete production integrations. ADK package metadata checked for this series reported 2.12.0; pin and test the version used by your application.

• Anthropic: many-shot jailbreaking (2024)
• OWASP authorization
• OWASP logging