Design ambient quality as a controlled regression discovery loop

Pattern 15 companion: Python example · Behavioral tests · Example license

An observation is the start of engineering work

An incident-response assistant announces that a service has recovered, but the outcome record still shows pending remediation. A quality observer notices the contradiction after the customer interaction. That finding is useful only if an engineer can inspect the supporting evidence and reproduce the behavior. This original design connects observation to a reviewed regression fixture rather than automatically turning every diagnosis into a production edit.

The broader Evals collection explains ambient quality and Google's AQuA implementation. This pattern focuses on the next boundary: evidence-backed regression discovery and controlled improvement. It is a separate original standard-library example, not an AQuA adapter or a copy of its implementation. It does not need a language model to demonstrate the ownership and acceptance contracts.

Architecture and request sequence

Component diagram for Design ambient quality as a controlled regression discovery loop — Original reference architecture by Sandeep Belgavi. Arrows show the responsibilities and transitions described below.

Request sequence for Design ambient quality as a controlled regression discovery loop — Illustrative sequence for this design; failure and recovery variants are explained in the article.

Keep serving, observing, and releasing separate

The serving agent handles the incident under its existing authorization and tool limits. The observer reads approved evidence and creates findings in a separate store. A reviewer decides which findings become sanitized regression fixtures. A release service evaluates candidate repairs and owns deployment approval. These components can cooperate without sharing all credentials or giving the observer permission to modify the system it measures.

OpenTelemetry describes traces as records of an operation's path through spans, with context connecting related work. Source: trace concepts. Correlation helps locate evidence, but it does not establish that the application outcome was correct. The proposed design links a session to a versioned outcome record instead of treating trace completion as business success.

Resolve evidence before judging the claim

Each synthetic session contains an identifier, revision, structured claim, and evidence reference. The evidence record contains its own reference, revision, outcome, completeness flag, and redaction flag. The observer resolves the reference in an approved local index and checks matching identity and revision. Missing or incomplete evidence produces unknown, not a pass and not a confirmed defect.

This is a stronger locator check than merely recognizing a reference-shaped string, but it remains a narrow deterministic rule. A completion claim conflicts when the recorded outcome is pending or failed. The example does not interpret arbitrary incident prose or identify root causes. A future model adapter can propose classifications after evidence validation, while deterministic permission and completeness checks remain outside the model's judgment.

Deduplicate without concealing contradictory records

The local observer identifies a session by its identifier and revision. Repeated identical records increment a duplicate count and do not multiply findings. Different content under the same identity increments a conflict count without overwriting the first record. This makes inconsistent ingestion visible. In a production stream, the identity scheme should also account for event generation, corrections, and deliberate reprocessing.

A finding count is not the same as a failure rate. A single incident can produce many repeated observations, while a large unobserved population produces none. Define the unit of analysis before presenting metrics. The example maintains process-local dictionaries and is not a durable streaming system. A production implementation needs persisted ingestion identity and explicit rules for rechecking unknown cases when new evidence arrives.

Promote a fixture through a separate review boundary

A fixture export requires independent approval and evidence marked redacted. Both flags are trusted inputs in the exercise, not automated proof of review or successful anonymization. The fixture stores the exact session and evidence used for the finding so a later test can reproduce the contradiction. No repair is generated and no deployment action is available in the module.

For the proposed engineering process, a reviewer checks the label, verifies sanitization, records the expected outcome, and assigns a regression owner. Candidate code changes run against this case and unaffected cases. Keep the discovery set separate from a locked acceptance set when tuning the observer itself. Otherwise a rule can look excellent simply because it was rewritten to recognize every case used to judge it.

Capacity and measurement assumptions must remain visible

Assume an approved daily slice of one thousand incident sessions for planning, not a claim about measured throughput. Track selected, fetched, complete, judged, unknown, duplicate, conflicting, and reviewed records. Bound batch size and investigation cost before adding a model judge. Human review capacity can become the bottleneck even when machine inspection is cheap, so prioritize findings with evidence and actionable ownership.

Precision measures the fraction of flagged cases confirmed as failures; recall requires labeled failures beyond the flagged set. Source: precision and recall definitions. The design's acceptance criterion is a defensible path from observation to a testable repair, with unknowns and sampling limits retained. It is not a promise that an observer detects every regression.