Operate independent verification as a measurable service

Pattern 03 companion: Python example · Behavioral tests · Example license

Track the reasons a plan fails

Separate plan rejection from execution uncertainty and verification disagreement. An oversized plan suggests planning-policy drift; an unknown executor claim suggests missing or unexpected evidence; a mismatch between mappings suggests conflicting observations or a faulty claim. These conditions require different owners. A dashboard that counts all three as verification failures can lead the team to weaken the verifier when the actual problem is a broken collector.

Record accepted plan size, planner revision, evidence source revision and verifier revision. Preserve timestamps for each retrieval. During an incident, telemetry can change rapidly, so a disagreement may be temporally legitimate. A service that was degraded at the executor's observation time may be normal when verification runs. Decide whether verification checks historical support or current state, and make that choice explicit in the report rather than confusing the two questions.

Architecture and request sequence

Component diagram for Operate independent verification as a measurable service — Original reference architecture by Sandeep Belgavi. Arrows show the responsibilities and transitions described below.

Request sequence for Operate independent verification as a measurable service — Illustrative sequence for this design; failure and recovery variants are explained in the article.

Use adversarial plans and conflicting observations

Inject an empty plan, duplicate step keys, an over-limit plan and a step with an unavailable reference. The supplied tests demonstrate the deterministic rejection and unknown paths. Add a plan that tries to name a privileged action when integrating a real adapter. The parser and policy gate should reject it before any executor receives work. A plan that is beautifully written but structurally invalid should have no operational advantage over any other invalid input.

For verification, use conflicting evidence, stale evidence and evidence with deliberately mismatched service identity. The current module tests only conflicting values; the latter cases belong to an expanded contract. Keep negative test cases after each verifier change. A more permissive judge can raise its pass rate while reducing its ability to catch wrong claims. The goal is agreement with independently established labels and evidence, not agreement with the planner.

Prevent a verifier from becoming a rubber stamp

Give reviewers access to a sample of accepted and rejected cases. Reviewing only rejections hides false approvals. Label failure identification separately from root-cause attribution and proposed repair effectiveness. A correct degraded label does not prove that a particular deployment caused the incident. If a model verifier explains a causal story, require evidence appropriate to that claim rather than treating a valid source locator as a causal experiment.

Measure disagreement between independent reviewers and preserve ambiguous cases. When the team cannot label a case reliably, the system should not pretend its model verdict resolves the ambiguity. Introduce an unresolved category and a clear owner. A production process can still be useful if it narrows investigation questions without declaring a final answer. Resist incentives that require every planned investigation to become verified merely to improve completion statistics.

Bound replanning and control cost escalation

When a verifier rejects a report, the next action might be another observation, a revised plan or human escalation. Limit the number of these cycles and account for all of them under the original incident budget. Otherwise each rejected plan can create a new apparently independent job with a fresh allowance. The original three-step ceiling then ceases to bound anything meaningful. Record parent plan identity and cumulative cost across revisions.

Use deterministic checks for shape, scope and required evidence before paying for model assessment. Cache immutable evidence only when its capture time remains appropriate to the question. Do not cache a successful verdict across changed source revisions. Track the cost of rejected plans, verifier retries and reviewer handling alongside successful runs. A rising cost per supported finding can signal a poor planning strategy even when the verifier itself remains accurate.

Set boundaries for repair and release

A verification result should enter a review or release policy with a documented meaning. In the example, true means a narrow value agreement and remediation remains none. In a production assistant, an approved finding might authorize creation of a draft incident note, while a rollback still requires separate evidence and authority. Do not reuse one verified flag for both tasks. Define distinct permissions and acceptance criteria for diagnosis, communication and operational changes.

For rollout, start with read-only shadow investigations and compare findings with existing incident reviews. Require reproducible evidence links, bounded plans, visible unresolved outcomes and independent sampling of accepted cases. Practice disabling the verifier when its data source becomes unreliable while preserving the incident workflow. The fallback should route to human review, not silently accept everything. Keep enough audit data to reevaluate affected reports after the verifier is repaired.

Sources and operational limitations

The original behavioral tests establish only the fixture contract. The proposed calibration and operating controls extend the measurement ideas in Evals; they are not deployed features. No measured accuracy, independence guarantee or cloud cost is claimed. Real providers, tools and stateful environments require their own fault-injection and evaluation evidence.