Operate controlled improvement with coverage and repair evidence

Pattern 15 companion: Python example · Behavioral tests · Example license

Do not confuse a quiet observer with a healthy agent

The incident assistant produces no new findings overnight. Before calling that an improvement, check how many sessions were eligible, selected, fetched, and judged with complete evidence. A broken collector can produce the same quiet dashboard as a successful repair. Track unknown and conflicting records as first-class outcomes, with age and ownership. The local module's counters are a small demonstration of that separation.

Maintain event time and ingestion time so delayed evidence is assigned to the correct release and observation window. An incident that started before a prompt fix can appear after the fix in a processing dashboard. Without the original timestamps and version identity, the team can misattribute old behavior to the new candidate or wrongly declare a regression resolved.

Architecture and request sequence

Component diagram for Operate controlled improvement with coverage and repair evidence — Original reference architecture by Sandeep Belgavi. Arrows show the responsibilities and transitions described below.

Request sequence for Operate controlled improvement with coverage and repair evidence — Illustrative sequence for this design; failure and recovery variants are explained in the article.

Measure finding quality with honest denominators

For an original numerical illustration, an observer flags twenty sessions and independent review confirms fifteen. Precision among those reviewed flags is fifteen divided by twenty, or seventy-five percent. If the entire labeled population contains thirty real failures, recall is fifteen divided by thirty, or fifty percent. If unflagged sessions were never labeled, recall is unknown. These numbers are teaching examples, not measurements of this module or a vendor system.

The metric reference distinguishes these quantities. Also separate evidence-validity rate, failure-classification accuracy, and root-cause correctness. A finding can identify a real bad outcome while proposing the wrong repair. Report reviewer disagreements and sampling coverage rather than compressing every assessment into one reassuring quality score.

Inject missing and hostile evidence

The local tests cover missing references, incomplete records, mismatched revisions, and reused identifiers with changed contents. Extend a real integration with denied storage access, late arrivals, truncated traces, and malformed producer records. Assert that these cases remain unknown or rejected as designed rather than becoming successful judgments. Run the failure drills against the actual collector and evidence store when those components exist.

Treat retrieved text as untrusted evidence. A comment telling the observer to suppress a finding cannot grant permission to change its policy or write production code. If a model judge is introduced, test injected instructions in tool output, incident messages, and source comments. Keep the action boundary enforced in code and credentials; a model's reassuring explanation is not a security test.

Require a repair experiment after diagnosis

A reviewed finding enters the regression corpus with an expected outcome and sanitized evidence. An engineer proposes a repair and runs baseline and candidate against that case plus unaffected cases. A repair that simply stops the assistant from making any completion claim can remove one symptom while preventing useful incident resolution. Include both safety and task-success assertions so the candidate cannot pass by abandoning the task.

Use a locked acceptance set for final decisions, separate from the examples used to tune the observer or repair. Group related incidents to reduce near-duplicate leakage. If reviewers inspect acceptance failures and redesign the rule, treat those cases as newly consumed tuning data. Keep a new independent acceptance set rather than presenting repeated tuning as fresh evidence of generalization.

Observe the observer and bound its privileges

Track investigation latency, evidence-fetch errors, unknown rates, duplicate rates, conflict counts, review backlog, fixture exports, and model cost when a model exists. Limit concurrent investigations and maximum evidence volume. A quality worker should not overwhelm the telemetry store during the incident it is supposed to help explain. Provide a pause switch that stops new work while retaining existing findings and review history.

The observer's identity should read only approved evidence and write only its own outputs. The review service owns approval, the sanitization process owns export eligibility, and the release controller owns deployment. Log each transition with an evidence revision and actor identity. This separation creates an auditable improvement process even when the diagnostic proposal is wrong or the model encounters hostile text.

Accept improvement only after observing the repaired behavior

After an approved release, compare outcomes for the intended population while checking coverage and traffic mix. An absent finding without matching traffic is not proof that the fix worked. Preserve counterexamples and regression fixtures so the same failure can be detected after later prompt, model, or tool changes. Record residual uncertainty when the available evidence does not support a conclusion.

This completes the control loop: an observable incident produces a bounded finding, review produces a reproducible test, evaluation supports a release decision, and post-release observation checks the result. The Python module implements only evidence inspection and fixture admission. Its value is making those boundaries executable and understandable before adding a powerful model or production credentials, not claiming that automated improvement can safely supervise itself.