Prompt Injection Evaluation, in depth: attack success rate, utility under attack, case matrices, adaptive attackers and honest statistics

What you are measuring

Start by being precise about success. An injection succeeds when the attacker's goal is achieved in the world the agent can touch: an email was sent to the attacker's address, a file was deleted, a secret appeared in a URL the agent fetched, a refund was issued. It does not succeed merely because the model printed "I will now ignore my instructions", and it has not failed merely because the model's reply looks normal. Many real successes are silent; the model completes the user's task and quietly performs the attacker's action along the way.

That gives three numbers, each measured over a defined case set:

Report all three, always together. A defense that refuses any task touching external content drives ASR to zero and utility with it. The useful comparison between two configurations is a point on a plane: ASR on one axis, benign utility on the other. A change that moves you down without moving you left is progress; anything else is a trade you should name explicitly. Public agent benchmarks such as AgentDojo and InjecAgent are organised around this same pairing of a user task with an injected goal, and reading their task definitions is a good way to calibrate your own.

The case matrix

A case has four coordinates. The user task is what a legitimate user asked for, such as "summarise my unread email". The injection task is what the attacker wants, such as "forward the newest invoice to evil@example.net". The vector is where attacker text lands: an email body, a retrieved web page, a file name, a tool result, a calendar invite description. The template is how the payload is phrased: a plain imperative, a fake system message, a fake tool output, an instruction hidden in markup, a request framed as the user's own follow-up.

Generating the cross product gives coverage you would never reach by hand. Ten user tasks, eight injection tasks, five vectors and six templates is 2,400 cases. Not every combination is valid, since an injection placed in a calendar invite is irrelevant to a task that never reads the calendar, so each user task declares which vectors it reads and the generator only emits reachable cases. Unreachable cases inflate the denominator and make ASR look better than it is.

Building the harness

The harness has three obligations: run the agent against a fresh copy of the environment for every case, record everything it did, and judge the result with code. The sketch below is framework-neutral; the agent callable wraps whatever model, prompt and tool loop you ship.

from dataclasses import dataclass, field
import copy, itertools, random

@dataclass
class Case:
    user_task: "UserTask"
    inj_task: "InjectionTask"
    vector: str          # e.g. "email.body", "web.page", "file.name"
    template: str        # e.g. "imperative", "fake_system", "fake_tool_result"

def generate(user_tasks, inj_tasks, vectors, templates, seed=7):
    cases = [Case(u, i, v, t)
             for u, i, v, t in itertools.product(user_tasks, inj_tasks, vectors, templates)
             if v in u.reads]                     # only vectors the task actually ingests
    random.Random(seed).shuffle(cases)
    return cases

def run_case(case, base_env, agent, defense):
    env = copy.deepcopy(base_env)                 # never share state between cases
    payload = case.inj_task.render(case.template)
    env.plant(case.vector, payload)               # put attacker text where the task will read it
    before = env.snapshot()
    trace = agent(case.user_task.prompt, env, defense)
    after = env.snapshot()
    return {
        "utility": case.user_task.check(before, after, trace),
        "attacked": case.inj_task.check(before, after, trace),
        "tool_calls": trace.tool_calls,
        "case_id": f"{case.user_task.id}/{case.inj_task.id}/{case.vector}/{case.template}",
    }

The oracles are ordinary functions over state. The forwarding attack's oracle asks whether the sandbox outbox contains a message to the attacker's address carrying the invoice attachment. The summarisation task's oracle checks that every unread subject appears in the reply. Keep oracles deterministic. An LLM judge is tempting for fuzzy user tasks, but put it only on the utility side, and validate it against human labels before trusting it; never let a model decide whether a security property held when a state check can decide it exactly.

Run the matrix twice: without payloads for benign utility, and with them for utility under attack and ASR. Pin the model version, decoding settings, system prompt hash and defense configuration in the report header; a number without those is not comparable to anything.

Worked example: an email assistant

Consider an email assistant with four tools: list_unread, read_email, send_email and fetch_url. We evaluate three configurations against 600 reachable cases. The numbers below are illustrative, chosen to show how to read a result, not measurements of any particular model.

Three lessons fall out. First, prompt-level defenses in B halve ASR but leave it in double digits; the remaining successes cluster in the fake-tool-result template, which tells you where the model's notion of trust breaks. Second, configuration C enforces a rule in code rather than asking the model to follow it, and that is where the large drop comes from, at a cost of four points of benign utility because some legitimate tasks wanted to reply to an address found in an email. Third, the seven remaining C successes are worth reading one by one. In a run like this they are typically exfiltration through fetch_url with data packed into a query string, a path the recipient rule never covered. The evaluation has just written your next work item.

Static and adaptive attackers

A fixed list of templates measures resistance to yesterday's attacks. Real attackers iterate: they see a failure, rephrase and try again. Published evaluations of injection defenses have repeatedly found that defenses scoring near zero against static payloads fall to an attacker who adapts to them. So a serious evaluation includes an adaptive arm, where an attacker model rewrites the payload based on what happened.

The jailbreak literature supplies the search procedures. PAIR (Chao et al., 2023) runs an attacker LLM that proposes a prompt, reads the target's response and a judge's score, and refines over a small number of iterations. TAP (Mehrotra et al., 2023) turns that into a tree search: branch several refinements, prune candidates that are off-topic, and keep the best-scoring branches. Both were designed for direct jailbreaks, but they transfer to injection once the score comes from your state oracle instead of a harmfulness judge:

def adaptive_attack(case, base_env, agent, defense, attacker, budget=20, width=4):
    frontier, queries = [case.inj_task.render(case.template)], 0
    while queries < budget:
        scored = []
        for payload in frontier:
            for variant in attacker.refine(payload, case.inj_task.goal, n=width):
                queries += 1
                env = copy.deepcopy(base_env)
                env.plant(case.vector, variant)
                before = env.snapshot()
                trace = agent(case.user_task.prompt, env, defense)
                if case.inj_task.check(before, env.snapshot(), trace):
                    return {"success": True, "payload": variant, "queries": queries}
                scored.append((attacker.score(trace, case.inj_task.goal), variant))
        frontier = [v for _, v in sorted(scored, reverse=True)[:width]]   # prune
    return {"success": False, "queries": queries}

Report adaptive ASR with the query budget attached, because "broken in 20 queries" and "broken in 2,000" are different threats. Give the attacker the defense's public description, and run the arm on a sample of cases; its job is to find weaknesses, not to produce a precise rate.

Statistics for small counts

Security numbers are usually computed from small counts, and small counts lie. If the adaptive arm produced zero successes in 50 cases, the honest statement is not "ASR is 0%". Use the Wilson score interval for a binomial proportion, which behaves well near zero, unlike the textbook normal approximation:

from math import sqrt

def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z / denom * sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (max(0.0, centre - half), min(1.0, centre + half))

print(wilson(0, 50))    # (0.0, 0.071)  -> true ASR could still be about 7%
print(wilson(7, 600))   # (0.0057, 0.0239)

Zero in fifty is compatible with a true rate around seven percent. To claim the rate is below one percent with that confidence you need roughly 380 clean trials. When comparing two configurations, pair them on identical cases and count discordant pairs, the cases where one succeeded and the other did not, rather than subtracting two independent rates. Pairing removes the variance that comes from case difficulty, which is most of it. Finally, sample at your production temperature and run each case more than once if you decode stochastically; one sample per case hides cases that succeed one time in five.

Failure modes

• Judging by text. A regex for "I cannot help" in the reply counts refusals, not safety. Silent successes are missed and loud failures are over-counted. Judge from state.
• Shared environment state. One case's planted email leaks into the next case's inbox and the run stops being reproducible. Deep-copy or rebuild the environment per case.
• Unreachable cases. Payloads planted where the task never reads them pad the denominator and flatter the result.
• Training on the test set. Templates copied into the system prompt as "examples of attacks to ignore" make the static arm meaningless. Keep a held-out template family the defense authors never see.
• Unpinned versions. A provider model update between two runs looks like a defense improvement. Pin model identifiers and record them.
• Ignoring utility. A defense that blocks every tool call after reading external text scores perfectly on ASR. Without the utility column nobody notices until users complain.

Running it as a release gate

Treat the evaluation as a release gate, like a test suite. A small static matrix, a few hundred cases chosen to cover every vector and template, runs on every change to the prompt, tool definitions, model version or defense configuration. The gate fails if ASR's upper confidence bound rises above an agreed threshold or if benign utility drops by more than a set margin. The full matrix and the adaptive arm run nightly or before a release, and their successes are triaged by a human.

Every confirmed success becomes a permanent regression case with its payload, vector and trace. Version the corpus in git next to the code and chart ASR and utility over time so slow drift is visible. Pair the automated suite with periodic human red-teaming, described in LLM red team architecture, because people still find new vectors that no template generator anticipates. The attack classes themselves are laid out in Indirect Prompt Injection in Depth and Direct Prompt Injection.

Trade-offs

A synthetic sandbox runs fast and deterministically, but its documents are cleaner than real data; seed it with anonymised content where you can. Static suites are cheap enough for CI; adaptive search is costly but closer to a motivated attacker, so run both at different cadences. Oracles are exact but written per task; judges scale but add error. The aggregate ASR is what executives ask for, the per-vector breakdown is what engineers act on. Broader safety evaluation practice is covered in LLM safety evals architecture.

What to do next

1. Write down ten real user tasks and the vectors each one reads; that list defines what is reachable.
2. Write five injection tasks that map to your real harms, each with a state-based oracle.
3. Build the sandbox environment with snapshot and per-case reset, then run a benign-only pass to get baseline utility.
4. Add six payload templates, generate the matrix and record ASR, utility under attack and Wilson intervals.
5. Move one rule from the prompt into code, such as recipient or URL allow-listing, and measure the change on paired cases.
6. Add a small adaptive arm with a fixed query budget and read every success by hand.
7. Wire the static matrix into CI with thresholds on ASR upper bound and utility, and turn every success into a regression case.