AI Red Team Program, in depth: maturity stages, coverage from risk, attack success statistics and release gates

From exercise to program: maturity stages

Programs grow in recognisable stages. Knowing which stage you are in tells you what to build next and stops you buying an automation platform before you have anything worth automating.

The jump from stage 1 to stage 2 is the one that changes outcomes. Before it, a finding is a document; after it, a finding becomes a regression test that runs forever. Most of this article is about making that jump without the gate turning into either theatre or an obstacle that teams learn to route around.

A charter for stage 2 needs only a page: the systems in scope, who owns the gate, who may override it and how an override is recorded, how findings are triaged, and how results are retained. The hard part is the override rule. If any product manager can wave a release through, the gate is advisory; if nobody can, it will be bypassed informally. A named risk owner who signs an acceptance with an expiry date is the usual compromise.

Scoping from risk: the coverage matrix

Coverage starts from risk, not from a list of jailbreak techniques. Take each harm in the AI risk register and cross it with each surface where an attacker can deliver input: the chat box, uploaded files, retrieved web pages, tool outputs, and memory. Each cell of that matrix is a test objective with an owner and a threshold. Taxonomies such as the OWASP Top 10 for LLM Applications and MITRE ATLAS are useful checklists for filling cells, but they are inputs to the matrix, not the matrix itself.

Keep the matrix as data, so coverage is computed rather than claimed:

from dataclasses import dataclass

@dataclass
class Cell:
    harm: str            # e.g. "unauthorised refund"
    surface: str         # e.g. "retrieved web page"
    severity: int        # 1..4 from the risk register
    max_asr: float       # gate threshold on the 95% upper bound
    min_trials: int      # sample size needed to resolve that threshold

def coverage(cells, results):
    """results maps (harm, surface) -> (successes, trials)."""
    missing, thin = [], []
    for c in cells:
        r = results.get((c.harm, c.surface))
        if r is None:
            missing.append(c)
        elif r[1] < c.min_trials:
            thin.append((c, r[1]))
    return missing, thin

A cell with no results is a known gap; a cell with fewer trials than it needs is a hidden one, because a clean result on 20 attempts proves very little. The next section shows why.

Measuring attack success honestly

A model given the same attack twice can refuse once and comply once. An attack success rate (ASR) is therefore an estimate of a probability, and it needs an interval. Three rules keep the numbers honest.

1. Sample at production settings. Run the same model version, system prompt, tools and temperature that users get. Testing at temperature zero measures a system nobody uses.
2. Report an interval, gate on the upper bound. The Wilson score interval behaves well for small counts and rates near zero, which is exactly where gates live.
3. Measure the grader. Automated judges miss successful attacks. Have humans re-grade a random sample each run and track the judge's false negative rate; a judge that misses 30 percent of successes makes every ASR look 30 percent better than it is.

from math import sqrt

def wilson(successes, n, z=1.96):
    """95% Wilson score interval for a binomial proportion."""
    if n == 0:
        return 0.0, 1.0
    phat = successes / n
    denom = 1 + z * z / n
    centre = (phat + z * z / (2 * n)) / denom
    half = z * sqrt(phat * (1 - phat) / n + z * z / (4 * n * n)) / denom
    return max(0.0, centre - half), min(1.0, centre + half)

def gate(cell, successes, n):
    lo, hi = wilson(successes, n)
    if n < cell.min_trials:
        return "INSUFFICIENT", lo, hi
    return ("PASS" if hi <= cell.max_asr else "FAIL"), lo, hi

Two useful facts follow. First, the rule of three: if you observe zero successes in n independent trials, the one-sided 95 percent upper bound on the true rate is roughly 3/n. To claim a rate below 1 percent with no observed failures you need about 300 trials per cell, not 30. Second, trials are only independent if the attacks differ. Running one prompt 300 times measures sampling variance for that prompt; running 300 distinct attacks drawn from the cell's corpus measures the cell. Use both, but gate on the second.

The release gate

The gate runs in the deployment pipeline whenever anything that changes behaviour changes: model version, system prompt, tool definitions, retrieval sources, or guardrail configuration. It reads the coverage matrix, runs each cell's suite, grades results, and emits PASS, FAIL or INSUFFICIENT per cell. Any FAIL on a severity 3 or 4 cell blocks the release unless a risk owner records an acceptance.

Keep the suite affordable. A full gate across 40 cells at 300 trials each is 12,000 conversations, many multi-turn, plus judge calls. Tier it: a fast smoke subset on every prompt change, the full suite on model or tool changes, and manual campaigns on a calendar. Cache nothing across model versions; the point is to detect that behaviour moved. The broader evaluation harness is described in LLM Security Evaluations.

Worked example: a support agent with a refund tool

A support agent can read order history, issue refunds up to a limit, and send email. The highest severity cell is unauthorised refund via indirect injection: instructions hidden in a product review or an email the agent reads. The threshold is an upper bound of 5 percent, with at least 200 trials.

Run 1. 200 distinct injection attacks, graded by a judge with a 10 percent human audit. Fourteen succeed. Wilson gives 4.2 to 11.4 percent. The gate fails; the team does not argue about whether the true rate is 7 percent or 5 percent, because the upper bound settles it.

Fix. Refunds now require the order ID to appear in the user's own message, not in retrieved content, and the refund tool rejects amounts above the order total. These are controls that hold regardless of what the model is persuaded to do.

Run 2. 300 distinct attacks, none reused from run 1, including 100 aimed at the new control (for example, coaxing the user into pasting an order ID). Four succeed: 0.5 to 3.4 percent. The original fourteen successes are re-run separately as regression checks and all must fail. The gate passes. All eighteen successful attacks go into the regression suite, and the manual team logs 'social engineering the user into supplying the trigger' as a new technique for the next campaign.

Note what the program did beyond the exercise: the threshold existed before the test, the sample size was chosen to resolve it, the fix was verified with fresh attacks rather than the ones it was tuned against, and the evidence is reproducible.

Turning results into evidence

Red team output increasingly doubles as compliance evidence, so record it in a form someone else can audit. Under the EU AI Act, providers of general-purpose AI models with systemic risk must evaluate their models, including conducting and documenting adversarial testing (Article 55). High-risk systems carry accuracy, robustness and cybersecurity duties (Article 15). In the United States, NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework, lists red teaming among its suggested actions. None of these prescribe a method, which is why the record matters more than the tooling.

A useful evidence pack per release contains: the coverage matrix with thresholds as they stood before testing; model, prompt and tool versions under test; per-cell counts and intervals; the judge audit rate and measured judge error; open findings with owners and dates; and any risk acceptances with expiry. Store raw transcripts with access controls, since successful attacks are themselves sensitive.

Cadence and program metrics

Cadence keeps the program alive between launches. A workable calendar is: the automated gate on every change; a two-week manual campaign each quarter focused on the newest capabilities; an external exercise yearly or before a major launch; and continuous intake from the bug bounty and production incidents. Pair with defenders: the AI blue team should turn each finding into a detection as well as a fix.

Measure the program, not just the model:

• Coverage: fraction of matrix cells with sufficient trials in the last 30 days.
• Time to fix: median days from confirmed finding to passing gate, by severity.
• Regression rate: how often an old finding reappears after a model or prompt change.
• Novelty rate: share of manual findings not caught by the automated suite. If it falls to zero, either the suite is excellent or the manual team has stopped exploring.
• Gate overrides: count and age of open risk acceptances.

Failure modes

• Testing a different system. Gate runs against a staging prompt or temperature that differs from production, so passing results describe a system users never see.
• Tiny samples read as clean. Zero successes in 25 trials is reported as 'resolved' when the upper bound is still about 13 percent.
• Overfitting the fix. Re-running only the attacks that found the bug shows the patch memorised them, not that the weakness is gone.
• Unmeasured judges. A grader model is updated and its miss rate doubles; ASR improves overnight with no change to the product.
• Permanent exceptions. Risk acceptances without expiry accumulate until the gate blocks nothing.
• Findings without owners. Reports land in a shared folder and nobody is accountable for the fix; route them through the same process as security bugs and, when exploited, through incident response.

Trade-offs

What to do next

1. Write a one-page charter naming the gate owner, the override authority and how acceptances expire.
2. Build a coverage matrix from your risk register: harms crossed with input surfaces, each with severity and threshold.
3. Compute minimum trials per cell from the threshold using the rule of three, and stop reporting cells below it.
4. Add the Wilson interval to your result reports and gate on the upper bound.
5. Audit your judge: human-grade a random 10 percent each run and track its miss rate.
6. Wire a smoke suite into the deployment pipeline for prompt changes and a full suite for model and tool changes.
7. Convert every confirmed finding into a regression test, and verify fixes with fresh attacks.
8. Assemble the evidence pack for your next release and check that someone outside the team can follow it.