AFP engineering series · Chapter 25 of 25
Combine threat modeling, guardrails, authorization, evaluation, rollout, and incident readiness into a staged deployment plan.
Make capability growth an explicit release decision
An assistant often begins as a read-only prototype and gradually gains retrieval, memory, tools, and external actions. Each addition changes the threat model. A successful chat evaluation does not justify adding payment authority, and a safe single-agent demo does not establish that delegated workers preserve the same restrictions.
Use capability stages with entry criteria and evidence. The goal is not to delay every feature; it is to make the additional risk and required controls visible. A team can move quickly when it knows exactly which invariant must be demonstrated before the next capability is enabled.
Stage one: a constrained read-only assistant
Start with authenticated sessions, tenant-scoped retrieval, source provenance, output handling, and bounded resource use. Use read-only credentials and a tool inventory that cannot mutate records. Test indirect injection and cross-tenant retrieval before introducing action tools.
Measure legitimate task completion, citation quality, unauthorized retrieval attempts, and output release behavior. Establish a baseline using representative tasks rather than only adversarial examples. The system must be useful enough that later security comparisons have a meaningful reference point.
Stage two: proposals without execution
Allow the assistant to prepare structured action proposals while a fake or disabled execution adapter records them. Validate schemas, resolve resources server-side, and evaluate policy decisions. Compare proposals with human-reviewed cases to identify missing evidence and common mistakes.
Shadow mode must not create effects. A tool named dry_run should be backed by credentials or an environment that cannot accidentally perform the real action. Record the proposed amount, destination, and authority evidence, but minimize sensitive data in routine traces. This stage reveals errors in model planning without exposing users to those effects.
Stage three: reviewed, bounded actions
| Requirement | Evidence before enabling |
|---|---|
| Exact-action approval | Changed arguments invalidate approval |
| Backend authorization | Direct API bypass attempts fail |
| Atomic resource constraints | Concurrent requests preserve invariant |
| Idempotent execution | Retries do not duplicate effects |
| Uncertainty handling | Processor timeout reconciles safely |
| Containment | Tool can be paused during active workload |
Enable a small population and narrow action limits, with clear ownership and support procedures. Revalidate permissions and balances after review. Keep the ability to revert to proposal-only mode without losing pending records or duplicating operations.
Use a canary release that compares both safety and usefulness. A low incident count in a tiny rollout is not strong evidence about rare failures, but it can reveal integration problems, confusing approval screens, or unexpected latency. Combine canary observations with the controlled evaluation suite.
Stage four: carefully expanded autonomy
Automation should increase only for well-understood, bounded operations with reliable evidence. Some actions may remain permanently reviewed. Do not use a general model confidence score as the sole condition for removing approval. Confidence can be miscalibrated and does not establish resource ownership or consent.
Evaluate the cumulative effect of many individually allowed actions. A per-refund limit does not prevent a large aggregate loss across repeated requests. Add tenant, account, and time-window constraints where the business model requires them, and monitor campaign patterns without treating every anomaly as malicious.
Maintain a release manifest
Illustrative python; adapt the persistence and identity adapters to your service.
release_manifest = {
"capability_stage": "reviewed_actions",
"model": "evaluated-model-id",
"sdk": "pinned-sdk-version",
"tool_inventory_digest": "reviewed-inventory-hash",
"policy_bundle": "afp-policy-7",
"evaluation_run": "controlled-suite-run-id",
"rollback_mode": "proposal_only",
"owner": "designated-service-owner",
}
# Store evidence links and known gaps with this record.A manifest should reference actual artifacts and test results, not placeholder values in a production release. Include configuration and data-transformation versions as well as code. A parser or retrieval change can alter the prompt-injection surface even when the model stays the same.
Review changes to tools, permissions, prompts, classifiers, and approval UI together. They form one effective system. A narrowly scoped code review can miss a dangerous combination, such as a new tool parameter paired with an old permissive policy default.
Define acceptance and rollback criteria
Hard invariants should fail the release: cross-tenant data exposure, unauthorized money movement, duplicate effects beyond the defined contract, or mandatory checks silently bypassed. Softer metrics such as false-positive burden and latency need explicit acceptable ranges and owners. Avoid averaging a severe security failure into an otherwise favorable quality score.
Rollback must preserve state. Reverting code while pending operations use a new schema can create another incident. Plan compatibility for queued work, held approvals, and uncertain processor outcomes. Test the rollback path in staging with synthetic operations in each state.
Worked release gate: adding automatic low-value refunds
Northstar proposes automatic refunds below a small product-defined limit. The existing system already supports reviewed refunds. Removing review changes the authority model, so the team does not treat it as a UI-only optimization. It defines an eligible subset using verified order state, original payment destination, current customer scope, and aggregate limits.
In shadow mode, the policy records which cases would qualify while human review remains mandatory. Compare those recommendations with completed investigations and legitimate outcomes. Inspect disagreements rather than assuming the human decision is always correct. Keep labels versioned as later evidence arrives.
Before enforcement, run concurrency, replay, destination-change, revoked-permission, and processor-timeout tests. Verify that the automatic path uses the same ledger and idempotency controls as the reviewed path. A separate fast endpoint that skips those controls would defeat the purpose of the earlier work.
Enable the capability for a limited population with a rapid switch back to proposal-only mode. Monitor aggregate refunds, confirmed fraud, legitimate completion, and uncertain operations. The per-action cap alone does not bound repeated loss, so include account and tenant windows appropriate to the business. Avoid presenting any example threshold as universally safe.
The rollback drill includes operations already reserved or uncertain. Turning off new automatic approvals should not erase the state needed to reconcile existing submissions. The release owner signs off on evidence links, unresolved limitations, and the containment procedure. This is the practical meaning of improving guardrails: autonomy increases only where the composed system has a clear authority contract, tested invariants, and a recovery path.
Keep improving after launch
Review incidents, appeals, detector disagreements, and legitimate failures regularly. Add new fixtures from observed weaknesses, retire stale assumptions, and re-evaluate whenever models or dependencies change. Keep original evidence separate from sanitized training or test material so privacy protections remain intact.
This series and its five labs provide a reference design and controlled exercises, not a certification of any deployed assistant. The practical endpoint is a service whose capabilities, policies, evidence, and recovery paths are explicit. AFP is effective when the organization can explain why an action was allowed, prove that denied actions had no effect, and contain failures without abandoning legitimate users.
Practice and continue
Open the interactive companion lab to inspect the assumptions and outcomes with synthetic data. Then return to the 25-chapter AFP learning path.
Sources and implementation scope
Primary references checked October 15, 2026. Framework details follow the linked documentation; the Northstar cases, policy contracts, and test designs are original illustrative designs. Python snippets labeled illustrative are not complete production integrations. ADK package metadata checked for this series reported 2.12.0; pin and test the version used by your application.