Evals / Harness engineering and ambient quality / 03

Inside AQuA: architecture, evidence, storage, and verification

Read Google's documented quality workflow alongside pinned implementation details, including a crucial verification default. Then design explicit boundaries for evidence, storage, and human-controlled repairs.

By Sandeep Belgavi · 22 October 2026 · Technical guide

A source-based conceptual view of AQuA
  1. 01Production trajectories and deployed source snapshots
  2. 02Sample and review observed sessions
  3. 03Cluster candidate findings
  4. 04Verify and track insights
  5. 05Optional diagnosis for an engineer to assess
Original conceptual drawing of the published workflow, not an official diagram or exact internal call graph. Verification verdict enforcement has a separate runtime setting.

Observation forms an outer loop around serving

Google's published AQuA workflow samples and reviews sessions, clusters findings, verifies candidates, and tracks insights over time. Optional diagnosis uses deployed source snapshots. The announcement warns that checking sampled cluster members does not validate every member of the cluster. Source: published workflow. The diagram above summarizes that account; it is not a statement that every installation executes an identical call graph.

Architecturally, the important separation is between a customer request and an investigation about customer requests. In an original shipment example, the serving agent creates or checks an order. The observer later examines the recorded interaction and its evidence. If the observer becomes unavailable, the application still needs its ordinary runtime protections. Observation should improve engineering feedback without becoming a hidden replacement for authorization, input validation, or transaction integrity in the request path.

Read the entry point before inventing an integration

The pinned entry point constructs an ADK App with an orchestrator root agent and exposes conversational orchestration through :streamQuery and investigation execution through :asyncQuery. Source: application entry point. Those names identify interfaces in the reviewed implementation; they are not a complete authentication or client invocation recipe. Use the accompanying deployment and CLI documentation for an actual connection.

For integration planning, write down the owner of each boundary: telemetry ingestion, investigation execution, insight access, and repair approval. A single 'agent' box obscures different permissions and failure modes. An ingestion problem calls for transport and identity checks. A poor diagnosis calls for evidence and rubric review. A stalled dashboard calls for access and presentation checks. Keeping these categories separate makes incidents easier to route and avoids solving a missing-data problem by repeatedly changing a model prompt.

Verification runs, but rejection is not enforced by default

The v0.1.0 implementation runs verification by default while leaving negative-verdict enforcement off by default. In the pinned configuration, is_verification_enforced(None) returns false. The code explains that missing tool-error evidence can otherwise discard genuine defects. Runtime defaults include a maximum of 1,000 sampled records, a seven-day lookback, 14-day inactivity auto-resolution, and a 900-second post-update delay. These are version-specific settings, not recommendations for every workload. Source: runtime configuration.

This distinction changes how a reviewer should read the output. A candidate that survives the pipeline has not necessarily passed a mandatory rejection gate. Preserve the verdict, the evidence available to the verifier, and the configuration that governed the run. Conversely, a negative verdict is not proof that the original production problem never happened. If the evidence is incomplete, both the finding and its rejection can need investigation. A verification stage provides another assessment; the team's acceptance process must decide what that assessment authorizes.

Missing tool errors change what the observer can know

The configuration template warns that the cloud_logging path does not capture raised tool errors in correlated traces and that some content requires access to referenced Cloud Storage objects. Its illustrative settings differ from runtime defaults; a template value is not automatically a default. Source: configuration template.

Suppose the shipment tool raises an exception but the collected conversation contains only the final apology. A reviewer can observe the apology and perhaps an absent receipt. It cannot safely reconstruct the omitted exception details. In this original operating design, an evidence-completeness field records that gap and routes the case for enrichment. Missing access should not silently become an empty successful fetch. A useful dashboard distinguishes no matching events, fetch denied, referenced object unavailable, and investigation completed without a finding.

Treat source and metric storage as security boundaries

The pinned configuration identifies a BigQuery dataset holding insights and insight occurrences. It keys source snapshots by observed-agent name and revision. It also states that published metric code executes with AQuA's credentials, making write access to that code's bucket security-sensitive. Source: snapshot identity and metric execution. Do not infer that a bucket holding quality-related material is harmless merely because its purpose is observation.

Our proposed application layout separates raw telemetry, sanitized investigation inputs, immutable source snapshots, finding records, and approved regression fixtures. This is a design recommendation, not a claim about AQuA's exact physical storage schema. Each category has a retention owner and an access policy. A code citation should resolve against the recorded revision, not today's main branch. A line number that exists establishes a valid locator; it does not establish that the line caused the failure. Keep evidence validation and causal diagnosis as separate review questions.

For a concrete access review, consider a developer who can upload source snapshots but cannot deploy the application. That permission should not accidentally allow the developer to replace executable metric code running under a stronger identity. Inventory code-bearing objects separately from passive evidence. Test denied access as deliberately as successful access. Document who can change each object category and how an investigation records the version it consumed. These are original review questions for the deployment team, not a claim that the repository automatically enforces this proposed separation.

Sources and the boundary between observation and repair

The launch article supplies the high-level workflow; the pinned entry point, configuration, and template supply the implementation details above. Read the configuration when a simplified architectural description appears to promise stronger enforcement. This article does not assert a transaction model, retention guarantee, or production accuracy because the reviewed evidence does not establish those claims.

In a practical release process, a finding becomes a proposed test before it becomes a production change. An engineer checks evidence, reproduces the behavior, and evaluates the candidate repair against unaffected cases. The release controller then applies the existing approval and rollback policy. An insight that disappears after a period without matching traffic is not, by itself, proof of a fix. Record traffic coverage and event times so 'not observed' remains distinguishable from 'corrected and retested.'