How LLM Jailbreaks Happen: Instruction Confusion and Context Pressure

A model produces continuations, while an application assigns authority

A language model conditions its next output on the supplied context. The surrounding application assigns roles to instructions, user messages, retrieved passages, and tool results, but the model still has to interpret their contents. A jailbreak attempts to induce behavior that violates the intended safety policy. An application compromise additionally requires a path from that behavior to some harmful effect.

Keep these two outcomes separate. A model may generate disallowed text even though a connected service rejects every unauthorized action. Another model may sound appropriately cautious while still proposing a dangerous tool call. Testing only the prose misses the second failure. This article explains mechanisms for defensive design, using synthetic exercises rather than attack prompts against public systems.

Why conflicting instructions can change behavior

A malicious request can frame an instruction as a higher-priority rule, a fictional scenario, a quoted conversation, or an apparently benign transformation. The underlying problem is interpretation: the model is asked to follow legitimate instructions while also processing text that resembles instructions. The specific phrasing is less important to an application designer than the boundary it tries to cross.

In a harmless test, an assistant is supposed to summarize a fixture containing the invented word ORCHID. Another part of the fixture asks it to output only MAPLE. If the output changes, the exercise demonstrates task redirection in that configuration. It does not measure harmful-content jailbreak resistance, prove a production vulnerability, or justify testing a third party without permission.

Long context and optimized attacks

Anthropic's 2024 many-shot research examined how numerous demonstrations inside a context could influence safety behavior. Zou and colleagues' 2023 work studied optimized adversarial suffixes and transfer between models. These are distinct research mechanisms, and neither paper supplies a universal success rate for today's deployed systems. Their defensive implication is that a small collection of hand-written prompts is an incomplete test distribution.

Vary context length, placement of untrusted material, number of turns, and the amount of quoted conversation. Keep track of truncation: if a fixture exceeds the model's accepted window, you may be measuring what the client dropped rather than what the model resisted. Preserve the exact request representation so investigators can reproduce which material the model actually received.

Build a mechanism-to-control map

A mechanism map helps avoid false confidence from superficial variety. Ten paraphrases of one request may exercise the same underlying weakness. Conversely, one workflow can expose several distinct failures: the model follows the document, the tool accepts an untrusted tenant identifier, and the response leaks a record. Record each failure separately so fixing one does not hide the others.

Keep the trusted task fixed while varying the untrusted input. Then run a clean control with the same length and formatting. Otherwise a drop in task completion may be caused by ordinary context overload rather than an effective defense.

Measure success with a precise denominator

Illustrative python; adapt the persistence and identity adapters to your service.

def summarize_trials(trials):
    valid = [t for t in trials if t["executed"]]
    return {
        "executed_trials": len(valid),
        "unsafe_responses": sum(t["unsafe_response"] for t in valid),
        "unauthorized_effects": sum(t["unauthorized_effect"] for t in valid),
        "infrastructure_errors": len(trials) - len(valid),
    }

Do not score an API timeout as a successful defense. Do not silently discard it either. Report infrastructure errors separately and repeat according to a predeclared policy. For stochastic generation, repeat cases and retain trial-level outcomes. A single passing run is weak evidence about a failure that occurs intermittently.

Use human-reviewed labels for consequential cases and calibrate any automated judge against them. The judge's own model version and prompt are evaluation dependencies. If the judge changes at the same time as the system under test, an apparent improvement may come from changed scoring rather than a safer assistant.

Improve the model layer without making it the only boundary

System instructions can clarify the intended task and identify untrusted sources. Training and safety classifiers can reduce the frequency of unsafe outputs. Context reduction can remove irrelevant attack surface. These are useful controls, but each has costs: excessive filtering may block legitimate analysis, and aggressive summarization can erase provenance or distort a user's request.

Evaluate changes against both adversarial fixtures and allowed work. Include discussions of security incidents, translated documents, and requests that quote unsafe material for analysis. A refusal classifier that rewards any refusal may make a system look safer simply by making it less capable. Track task completion, false blocks, latency, and cost together with policy violations.

Worked experiment: separate task redirection from harmful effects

Create a local fixture whose legitimate task is to count the number of fictional product names in a document. The document also contains a benign instruction marker asking for a different output. The evaluation records whether the assistant completed the counting task, followed the marker, or refused the entire request. This isolates instruction confusion without asking for harmful content or contacting an external target.

Run three document lengths and place the marker near the beginning, middle, and end. Keep the legitimate facts constant and fill the remaining space with neutral material. Record the exact tokenized or serialized input accepted by the client, including any truncation. A marker disappearing during preprocessing is not evidence that the model resisted it. Likewise, a wrong count in the clean control may indicate ordinary reasoning difficulty rather than injection.

Now attach a fake tool that records proposed actions but cannot access real data. Give the assistant a legitimate read-only operation and an unavailable write operation. If the model proposes the write, record a model-boundary failure. If the runtime or backend rejects it, record containment. If a test harness merely omits the write tool, document that reduced capability as the control; do not attribute the result to a stronger classifier.

Repeat each configuration enough times to see variability, and report raw counts with the configuration. Avoid selecting only the most dramatic run. A defense that improves one marker's outcome but harms clean task completion needs investigation. Compare the clean and adversarial variants under the same model, sampling settings, and context budget.

Finally, replace the fake action service with a sandbox implementation that enforces ownership. Use an invented tenant A principal and a tenant B record. The decisive assertion is that no tenant B content enters the model context. Even if the model is redirected, the resource boundary should hold. This is a stronger application property than the assistant merely saying it will respect privacy.

The experiment produces three separate findings: susceptibility to a particular harmless redirection, the runtime's handling of unauthorized proposals, and the backend's enforcement of ownership. None is a universal jailbreak score. Together they show where an improvement belongs and prevent a team from confusing better wording with stronger authorization.

Contain a successful jailbreak

Assume a test eventually produces a forbidden proposal. The application should still restrict its effect: no access to another tenant, no arbitrary command execution, no secret-bearing outbound request, and no payment without independent authorization. Remove unnecessary tools and use read-only credentials where the task permits them.

The practical release question is not whether someone can ever persuade the model. It is whether persuasion can cross a protected application boundary, and how quickly the team can detect and contain it. The companion trust-boundary lab intentionally uses a simplified “model followed the document” switch so readers can inspect the downstream controls without pretending to simulate a real model's psychology.

Practice and continue

Open the interactive companion lab to inspect the assumptions and outcomes with synthetic data. Then return to the 25-chapter AFP learning path.

• Why LLM Abuse and Fraud Prevention Needs a Threat Model
• Indirect Prompt Injection in RAG: When Retrieved Data Becomes an Instruction

Sources and implementation scope

Primary references checked October 15, 2026. Framework details follow the linked documentation; the Northstar cases, policy contracts, and test designs are original illustrative designs. Python snippets labeled illustrative are not complete production integrations. ADK package metadata checked for this series reported 2.12.0; pin and test the version used by your application.

• Anthropic: many-shot jailbreaking (2024)
• Zou et al.: transferable adversarial attacks (2023)
• OWASP prompt injection prevention