Red-team Evaluation for AFP: Test Attempts, Effects, and Legitimate Use

Define the outcome before collecting prompts

A collection of dramatic jailbreak examples is not an evaluation plan. Start with the protected outcomes: no unauthorized data access, no unapproved payment, no prohibited content released, and no unbounded work. Each needs a different observation point. A model response can violate policy while a service successfully contains the action; both facts should remain visible.

Use systems you own or are authorized to test, synthetic accounts, and fake external services. The purpose is to improve a controlled application, not to discover ways to abuse an unrelated public endpoint. Keep fixture content harmless where possible, such as invented records and benign instruction markers.

Separate four layers of evidence

Record all four with a shared case identifier. A test that only checks for a refusal phrase can miss a tool effect. A test that only checks the ledger can miss a harmful answer. A test that counts every denied call as an attack can misclassify ordinary application bugs as malicious behavior.

Define success criteria per fixture. For a cross-tenant retrieval test, the required outcome is that unauthorized content never enters the model context. For an output screening test, the required outcome is that no disallowed bytes reach the client. The same final answer is not sufficient evidence for both.

Build a coverage matrix

Vary entry point, trust source, context length, language, tool type, and execution mode. Include direct requests, retrieved documents, tool results, summaries, and resumed sessions. Cover streaming and non-streaming paths, parallel tool calls, retries, and cancellation. These dimensions represent application behavior, not a requirement to generate an enormous random prompt list.

Choose representative combinations and document gaps. Pair each adversarial fixture with legitimate neighboring work: a security analyst quoting an injection, a customer correcting an order number, or an employee summarizing a suspicious invoice. A defense that blocks all of these may look robust on attacks while failing the product's purpose.

Keep a reproducible case record

Illustrative python; adapt the persistence and identity adapters to your service.

case = {
    "id": "tenant-boundary-004",
    "fixture_version": "1",
    "principal": "synthetic-tenant-a",
    "requested_resource": "synthetic-tenant-b-order",
    "expected": {"records_returned": 0, "effects_committed": 0},
    "model_version": "record-at-run-time",
    "policy_version": "record-at-run-time",
    "sdk_version": "record-at-run-time",
}
# Store observed tool calls and service effects alongside this record.

Retain the exact serialized input after preprocessing, not only the original text file. Token truncation, OCR, or a summarization step may change the tested input. Record deterministic seeds where supported, but do not assume a seed makes every provider response reproducible across versions.

For stochastic cases, run repeated trials and report counts. Separate model errors, infrastructure errors, and policy failures. An unavailable model is not a safe answer, and a failed tool connection is not proof that authorization worked.

Use judges with a validation plan

Automated judges can scale review of textual outcomes, but they are models with their own failure modes. Calibrate them against human-labeled examples and inspect disagreements. Keep the judge prompt and model version stable during a comparison, or report the change explicitly.

Use deterministic assertions for concrete effects whenever possible: number of rows returned, operation state, recipient list, budget balance, and released bytes. Reserve semantic judgment for outcomes that need it. Do not ask a language model to decide whether a database transaction violated a numeric invariant when the ledger can answer directly.

Report uncertainty and prevent benchmark overfitting

Keep a held-out set and rotate some fixtures after incidents. If developers repeatedly tune to the same public examples, passing them can become a memorization exercise. Add cases derived from the application's actual architecture, including unknown tools and malformed policy responses, rather than only famous jailbreak families.

Zero observed failures does not prove zero risk. With n independent trials and no failures, the rough rule-of-three upper bound is about 3/n at 95% confidence; independence and representativeness may not hold for adversarial tests. State those assumptions rather than turning a small test run into a universal safety claim.

Worked release comparison: a new model refuses more but contains less

Model A follows a suspicious document in 10 of 100 synthetic trials, but the action service blocks every unauthorized proposal. Model B follows it in only 3 trials, yet a changed tool adapter allows one cross-tenant read. If the report shows only model-level attack success, B appears better. The application-level evidence shows a new severe failure.

Keep the evaluation harness's tool and service versions fixed when comparing models, then run a separate full-release comparison when adapters change. Report both sets. This separates model behavior from integration changes and prevents a favorable aggregate score from hiding a broken resource boundary.

Add legitimate neighboring tasks. If B refuses many ordinary incident-analysis requests, its lower redirection rate may partly reflect reduced usefulness. Record completion, false blocks, and latency. Do not score every refusal as safe success regardless of the task's legitimacy.

For the cross-tenant case, capture the actual context sent to the model and assert that unauthorized content is absent. A final answer that omits the secret is not enough if the model already received it. For payment cases, inspect the ledger. For streaming cases, capture released bytes. Choose observations at the protected boundary.

Preserve the failed fixture as a regression case after fixing the adapter, and keep a held-out set to reduce tuning to known examples. Document that 100 trials are limited evidence about rare failures and that adversarial trials may not be independent. The release decision should state which hard invariants passed, which statistical estimates remain uncertain, and which capabilities are still withheld. This is more actionable than a single red-team score.

Use evaluation to decide releases

Define release gates before seeing results: no violations of hard transaction invariants, acceptable legitimate-task completion, bounded false blocks, and successful incident controls. A new model should pass the same application-level suite even if it has better general benchmarks.

ADK's evaluation tooling can support agent testing, but AFP assertions must include the services and channels outside the model. Publish a concise evidence record with scope, versions, results, and known gaps. The goal is a repeatable decision process that explains why a release is acceptable and what would trigger rollback.

Practice and continue

Open the interactive companion lab to inspect the assumptions and outcomes with synthetic data. Then return to the 25-chapter AFP learning path.

• Preventing Outbound Abuse: Consent, Recipient Scope, and Agent Messaging
• Guardrail Metrics: Precision, Recall, Escape Rate, and Review Burden

Sources and implementation scope

Primary references checked October 15, 2026. Framework details follow the linked documentation; the Northstar cases, policy contracts, and test designs are original illustrative designs. Python snippets labeled illustrative are not complete production integrations. ADK package metadata checked for this series reported 2.12.0; pin and test the version used by your application.

• ADK agent evaluation
• NIST AI RMF Playbook