A prompt change is a production change
The incident-response assistant's code is unchanged, but a new prompt encourages immediate remediation instead of evidence gathering. The same container image can now behave differently. A release identifier based only on the application commit cannot explain the change. This original design treats code, prompt, model selection, tool schema, and state schema as a bundle whose identity is recorded with every run.
The bundle is an inventory of versioned inputs, not a guarantee of deterministic model behavior. A hosted provider can have behavior outside the application's control, so retain the provider's documented version identifiers and available response metadata. Record retrieval and policy revisions where they affect the task. The teaching bundle uses five short synthetic strings to make the identity boundary executable without installing a provider SDK.
Architecture and request sequence
Read the architecture as text
- Immutable release bundle: Code, prompt, model, tools, schema
- Cohort assignment: Stable tenant and incident hash
- Cohort observations: Outcomes and collection coverage
- Release decision gate: Hold, approve, promote or rollback
- Deployment adapter seam: Actual rollout requires external adapter
- Immutable release bundle → Cohort assignment: Candidate identity
- Cohort assignment → Cohort observations: Assigned traffic
- Cohort observations → Release decision gate: Validated observations
- Immutable release bundle → Release decision gate: Compatibility
- Release decision gate → Deployment adapter seam: Authorized decision
Read the sequence as text
- Release → Gate: Submit candidate bundle
- Gate → Metrics: Request cohort evidence
- Metrics → Gate: Counts and hard violations
- Gate → Release: Hold or proposed decision
- Release → Gate: Bound approval
- Gate → Platform: Authorized desired state
Separate artifact identity from rollout policy
An immutable candidate should remain the same artifact as it moves from evaluation to canary to wider release. The percentage of traffic assigned to it belongs to rollout policy, not to the candidate's source identity. Changing a prompt after a passing evaluation creates a different candidate and requires new evidence. An operator should be able to reconstruct which artifact and which policy governed a particular incident.
The example hashes the five bundle fields with SHA-256. This provides a stable identifier for those supplied strings; it does not verify a container signature or prove that the strings accurately name deployed artifacts. A production release service needs provenance and access control around artifact creation. Keep those stronger claims separate from the local demonstration's deterministic identity calculation.
Choose cohorts that preserve a coherent incident
Randomly routing each message can send one incident through several incompatible prompt or tool versions. Our proposed service assigns a cohort using tenant and incident identity, keeping the selection stable for a given canary policy. The local example hashes that pair into a bucket and compares it with a percentage. It demonstrates stable assignment, not statistically balanced sampling or deployment routing.
For long incidents, define what happens when the rollout percentage changes or the candidate is withdrawn. Existing runs might stay pinned, pause for migration, or restart under an approved policy. The choice depends on state compatibility and the risk of continuing the candidate. A stable hash alone cannot define this lifecycle. Store the chosen release in the run record when stability across policy changes is required.
Canary evidence needs a control and enough observations
Google SRE describes canaries as limited deployments evaluated against a control and emphasizes representative measurements. Source: canarying releases. In our original gate, fewer than twenty observations in either cohort yields hold. A candidate failure rate more than ten percentage points above control triggers rollback. These numbers are teaching thresholds, not statistically justified production defaults.
A serious acceptance policy needs a minimum observation window, sample adequacy, task mix, and business-impact tolerances. A rare destructive action cannot be averaged away by many harmless successes. The example therefore treats any reported authorization violation as an immediate rollback condition. Define similarly hard constraints for your application and keep them independent from average quality metrics.
Rollback needs state compatibility
Reverting an application bundle cannot automatically undo external actions or restore an incompatible database. The example conservatively requires baseline and candidate schema identifiers to match. That is a narrow guard, not a migration engine. A real expand-and-contract migration needs a compatibility matrix showing which versions can read and write each state form during rollout and rollback.
Kubernetes documents Deployment revisions and rollback behavior for the workload's pod template. Source: Deployment management. That platform mechanism does not establish application-level recovery for agent memory, tool effects, or external data. Treat workload rollback, state migration, and business compensation as separate operations with separate owners and tests.
Plan capacity for two versions and a pause
A canary can require baseline and candidate workers at the same time, plus evaluation and telemetry capacity. For this original planning assumption, reserve enough baseline capacity to serve the entire admitted workload if the candidate is removed. Measure that capacity with realistic tool latency rather than equating a worker count with successful incident throughput.
The release service should be able to pause new candidate admissions while retaining evidence and in-flight state. It also needs an authorized rollback path independent of the candidate model. The acceptance question is whether a release owner can identify the active version, explain the gate decision, and restore a compatible baseline under the tested failure. The following build article implements only the decision logic locally.