Understand jailbreaks, indirect prompt injection, safe evaluation, and the application controls that contain model failures.
Two failures that need different measurements
A jailbreak attempts to make a model violate its safety policy. Prompt injection is broader: attacker-controlled text tries to redirect the application away from its intended task. An assistant can refuse harmful questions yet still follow an instruction hidden inside a retrieved document. Conversely, an unsafe textual answer does not automatically imply that a connected database or payment service was compromised.
OWASP's prompt injection guidance distinguishes direct user input from indirect instructions in external material and notes that RAG does not eliminate this risk. Treat those as separate evaluation surfaces. The design examples below are engineering proposals, not claims that any particular model is immune.
Map the trust boundary before tuning the prompt
Imagine a support assistant that reads a customer's ticket, retrieves a refund policy, and proposes a refund. The ticket is a claim from the customer. The retrieved policy has an owner and version. The authenticated session establishes who is asking. The payment service establishes what can actually happen. Concatenating all four into a conversation must not make them equally authoritative.
Write down each input's origin, permitted use, and maximum effect. Uploaded documents can provide facts to summarize; they cannot grant access to another account. Search results can identify a candidate source; they cannot authorize an outbound network request carrying private conversation data. Preserve source identifiers when creating summaries so later agent steps do not mistake a compressed quotation for a trusted instruction.
Why long conversations belong in the test suite
Anthropic's April 2024 many-shot jailbreaking study found that repeated demonstrations in a long context could influence safety behavior in the models tested. It is historical research, not a measurement of every model available today. Its useful lesson for a product team is to test the context lengths and conversation patterns the product actually permits.
Include long document collections, multi-turn corrections, quoted conversations, multilingual inputs, and tool-result summaries. Do not assume a short single-turn benchmark covers an agent that runs for twenty steps. Keep test fixtures synthetic, with invented customer records and a harmless marker as the target output. Run only against systems you own or have permission to evaluate.
Enforce decisions outside the model
For the refund example, expose a narrow proposal tool rather than a general database executor. Resolve the account from the server-side session, fetch the order from the authoritative store, and validate eligibility using application code. The model may identify a candidate order, but it cannot select the tenant, expand its permissions, or create an approval token.
Separate proposal from execution. A preview should show the exact order, amount, currency, and destination. Bind approval to those values and expire it when the proposal changes. Recheck permissions at execution time: an approval granted before an account was suspended should not remain sufficient afterward. Use an idempotency key and transactional accounting so retries cannot issue duplicate refunds.
For read-only assistants, enforce document access before retrieval and before returning results. Redact secrets before they enter context. A refusal is useful behavior, but an access check is the control that prevents one customer's records from reaching another customer's session.
Evaluate outcomes, not refusal vocabulary
Measure policy-violating responses separately from unauthorized tool calls and actual unauthorized effects. A tool call rejected by the service is an application containment success even if model behavior still needs improvement. A polite refusal followed by a successful forbidden action is a failure.
Record the model version, context fixture, tool schema, policy version, and permitted outcome for every case. Repeat stochastic cases and report the number of trials. Include legitimate requests that resemble adversarial cases: a security researcher quoting an injection, a translated support request, or a document discussing fraud. Track false blocks alongside attack success so tightening a filter does not silently make the product unusable.
Operate with an explicit containment plan
Log the decision category and tool result with a request identifier, while limiting sensitive prompt retention. If a new attack succeeds, disable the affected capability, preserve a minimal reproducible fixture, and repair the missing boundary. Re-run both the regression case and legitimate neighboring tasks before restoring access.
The release criterion should describe an enforceable property: for example, no request can read another tenant's order and no refund can execute without a matching approval. Avoid claiming that a new system prompt makes jailbreaking impossible. Prompts, classifiers, and model training can reduce failures; service permissions determine their consequences.