Begin with an explicit uncertainty queue
An operational dashboard should distinguish reserved, compensated, rejected, and unknown outcomes. The local module has no remote uncertainty because both effect and receipt share one transaction. A production remote integration does. When a response disappears after dispatch, place the operation in a reconciliation queue rather than reporting failure as if nothing happened. The assistant can explain that confirmation is pending and expose an operation reference.
An operator should be able to inspect the original intent, authorization decision, dispatch attempts, and provider receipts. Do not make the operator reconstruct this history from a fluent model transcript. Unknown operations need an owner, an age threshold, and a documented way to query the external system. Those policies prevent retries from becoming the default response to every ambiguous event.
Architecture and request sequence
Read the architecture as text
- Dispatch observations: New, retry, conflict and unknown
- Reconciliation queue: Inspect uncertain remote outcomes
- Operator decision: Authorize bounded recovery
- Outcome history: Retain effect and compensation evidence
- Dispatch observations → Reconciliation queue: Classified signal
- Reconciliation queue → Operator decision: Reviewed evidence
- Operator decision → Outcome history: Controlled response
Read the sequence as text
- Runtime → Monitor: Report scoped observations
- Monitor → Review: Check coverage and failures
- Review → Operator: Submit evidence for review
- Operator → Runtime: Authorize bounded next step
- Runtime → Monitor: Record observed outcome
Inject faults at the real boundaries
The companion test injects failure between the local hold and receipt writes. Extend integration testing with a lost acknowledgment after the external service commits, a worker restart before receipt persistence, and a duplicate delivery after compensation. Assert the business effect count, not just an HTTP status. A tool can return an error after committing work, so an error-only assertion can miss the incident you meant to prevent.
Test conflicting arguments under the same key and a delayed retry after the configured receipt-retention period. The latter forces a product decision: reject an expired operation reference, require reconciliation, or deliberately permit new work under a new intent. An implementation that silently forgets the first effect changes its semantics when time passes. Document that behavior before scheduling automated cleanup.
Keep cleanup permissions narrower than execution
The actor allowed to release a maintenance hold should not automatically have every permission used to create it. Define compensation as its own authorized operation, with the original operation reference and an operator reason. Validate tenant and service ownership again at that boundary. A transcript that says the incident is resolved is evidence to review, not authority to release a production protection.
Audit access to receipts because they can reveal service identifiers and incident timing. Minimize secrets in request fingerprints and log fields. The educational module uses a boolean permission seam to expose ordering; a production service needs verified identity and policy evaluation. Tests should ensure a denied action leaves both effect and receipt unchanged, including after a previous successful operation.
Measure effects and queue age rather than retry volume alone
Track new intents, matching retries, conflicting retries, completed compensations, compensation failures, and reconciliation age. A rising retry count with stable unique effects can mean the idempotency boundary is working while a network path is failing. A rising conflict count can indicate a caller regenerating arguments under reused identifiers. Those situations need different responses. Attach tenant-safe correlation identifiers without placing full incident descriptions in high-cardinality metric labels.
Measure storage contention and transaction latency separately from model latency. If the gateway serializes on a hot service, adding more planning agents can increase pressure without increasing useful throughput. Use sampled traces to connect proposal, authorization, transaction, and result. Keep durable operation receipts as the source of business state rather than treating an observability backend as the ledger.
Budget for reconciliation and retention
An illustrative cost model separates receipt writes, receipt reads for retries, retained audit bytes, and human reconciliation time. Ten cheap retries are not necessarily more costly than one ambiguous production effect requiring an engineer. Avoid optimizing away retention before measuring the delay distribution of real retries and incident investigations. Archive sensitive history according to a deliberate policy and test the lookup behavior after archival.
Compensation can require additional capacity during the same incident that saturated execution. Reserve a bounded worker pool and monitor its oldest item. If a compensation cannot complete, report partial recovery with the original effect still visible. Do not mark the entire workflow successful merely because the compensation job was enqueued. Queue acceptance and business completion are separate outcomes.
Acceptance requires a repeatable recovery story
Before enabling automatic action retries, demonstrate one effect under duplicate delivery, rejection of changed intent, recovery after restart, and accurate reporting of an unresolved remote outcome. Before enabling automatic compensation, demonstrate repeat-safe cleanup and an escalation path when reversal fails. The tests in this module cover only the local subset; a production acceptance record must identify the additional adapter tests and their environment.
AWS's saga discussion treats compensation and observability as explicit distributed-workflow concerns. Our operating recommendations extend that principle to an incident assistant without claiming distributed atomicity. The final decision is whether an operator can explain the system's current effect after the worst tested interruption, not whether the model produced a confident completion message.