Operate agent releases with rollback evidence and versioned telemetry

Pattern 14 companion: Python example · Behavioral tests · Example license

Watch the release as a system change

An incident assistant can regress through a prompt edit, a model revision, a tool schema change, or a retrieval-policy update. Attach the complete release identifier to runs and findings so the operator can separate these changes. A dashboard grouped only by container version can conceal a prompt rollout. Conversely, changing several independent components at once makes a negative result harder to attribute.

For this original operating model, release one reviewable bundle with a recorded change hypothesis and expected outcomes. Preserve the baseline evidence and candidate evidence using the same acceptance set where appropriate. If the candidate changes during evaluation, create a new identity rather than editing the old result. The release record should explain what was evaluated and what was actually deployed.

Architecture and request sequence

Component diagram for Operate agent releases with rollback evidence and versioned telemetry — Original reference architecture by Sandeep Belgavi. Arrows show the responsibilities and transitions described below.

Request sequence for Operate agent releases with rollback evidence and versioned telemetry — Illustrative sequence for this design; failure and recovery variants are explained in the article.

Observe both traffic quality and collection quality

Compare candidate and control outcomes, but also compare telemetry completeness, task mix, latency, and tool access. A candidate that drops difficult requests before they are counted can appear more accurate. Track admissions, completed analyses, failures, cancellations, and missing observations separately. Keep event times alongside collection times so a delayed metric does not become evidence about the wrong version.

The teaching gate uses supplied integer counts and cannot detect biased or stale input. A production metrics adapter must identify the cohort, observation window, query revision, and exclusion rules. If collection becomes unreliable, hold the release. No findings in an empty or broken observation window are not sufficient grounds for promotion, even when the deployment platform itself reports healthy processes.

Drill rollback before depending on it

Inject a synthetic hard violation after a candidate is active and verify the entire rollback path: decision, authorization, platform request, observed traffic shift, and post-rollback health. The local test verifies only the gate's active-bundle field. An actual drill must check the infrastructure and application state. Measure how long new candidate work continues after the stop decision.

Also test a failed rollback request and an incompatible state transition. Operators need an escalation procedure when the baseline cannot safely resume. External actions already performed by the candidate require reconciliation or compensation; replacing the running code does not erase them. Record these residual effects explicitly so a green deployment status does not imply that every business consequence has been repaired.

Keep release authority outside model judgment

A model-generated explanation that the candidate is safe should not grant deployment permission. Bind approval to the immutable candidate identifier, the accepted evidence, and the intended rollout scope. Expire or invalidate that approval when the candidate changes. Restrict who can alter gate thresholds and preserve the previous policy for audit and rollback analysis.

The same separation applies to telemetry queries. A candidate agent should not be able to edit the query that decides whether it passes. Review metric definitions and test malformed counts, missing cohorts, and hard-violation reporting. The example's integer validation is only a first boundary; a real integration needs trusted provenance and access controls for the underlying observations.

Budget for evidence and reserve recovery capacity

A release can temporarily duplicate workers and evaluation calls. Budget that overhead before starting rather than treating it as an unexpected production incident. Retain enough baseline capacity to absorb candidate traffic under the planned rollback. If the canary shares a saturated dependency with control, investigate whether the comparison reflects the candidate or the shared overload.

Google SRE's canary guidance emphasizes evaluation that supports a rollout decision. Our original extension records infrastructure cost, investigation cost, and operator time alongside quality. A fast promotion that saves observation cost can be expensive if it removes the evidence needed to detect a rare but severe regression. Choose observation windows from risk and traffic, not convenience alone.

Close the release with an auditable outcome

A completed release record should identify the bundle, cohort policy, evidence window, approval, platform result, and post-release checks. If the release is held or rolled back, retain the reason and the unresolved work. Do not overwrite a failed candidate's evidence with the next attempt's results. That history supplies future regression cases and explains why the gate exists.

Acceptance means the team can reproduce the decision from recorded inputs and demonstrate the recovery path at the infrastructure boundary. The standard-library module helps test decision rules without external cost. Production confidence comes from connecting those rules to trustworthy telemetry, authorized deployment, and compatible state. The next pattern uses the resulting incidents and findings to discover regressions without giving an observer unrestricted release control.