Recovery begins with classification
When an incident workflow stalls, classify its condition before restarting it. A provider may be unavailable, evidence may violate the schema, a version may be incompatible, or an external action may have an unknown result. These are different operational states. Treating every exception as a reason to retry can turn an input defect into a costly loop or repeat a side effect that already succeeded.
The reference has only two productive transitions and exposes their persisted state through status. Start by observing the run identifier, version, step and last accepted evidence. Add error category, attempt count and timestamps in a deployment. A recovery worker should use those records to choose a bounded action. It should not ask a model to guess which database writes survived from a human-readable log fragment.
Architecture and request sequence
Read the architecture as text
- Run inventory: Age and current step
- Recovery classifier: Retry, reconcile or escalate
- Version registry: Compatible execution paths
- Worker: Bounded continuation
- Receipt audit: Check business completion
- Run inventory → Recovery classifier: stalled run
- Recovery classifier → Version registry: check version
- Version registry → Worker: authorized resume
- Worker → Receipt audit: evidence
- Recovery classifier → Receipt audit: uncertain outcome
Read the sequence as text
- Monitor → Operator: Flag stale checkpoint
- Operator → Store: Inspect state and version
- Operator → Worker: Authorize bounded recovery
- Worker → Store: Commit next transition
- Operator → Store: Verify receipt
Rehearse process loss at each boundary
Use the tests to rehearse failure before either local commit. Then extend the exercise by terminating a separate worker process during a real adapter timeout, preserving the database and starting a new process. Verify that committed evidence remains stable and that uncommitted work is classified as repeatable or uncertain according to the adapter contract.
The fixture's read can safely repeat, so the test explicitly observes an extra provider call. A real diagnostic query may be read-only but still expensive or time-sensitive. Capture the observation timestamp and disclose that a retry can see a changed world. If the task requires the original snapshot, the provider must support that snapshot identity; a checkpoint cannot recreate a measurement that was never recorded.
Upgrade without reinterpreting history
The example rejects a persisted v1 run when opened as v2. Operationally, that failure should route to a compatibility decision rather than repeated worker restarts. Options include keeping old workers available, migrating a documented state representation, or cancelling and starting a new run with explicit linkage. Each choice has different effects on auditability and user expectations.
Before release, keep fixtures from in-flight states and run them against the proposed worker version. Verify both compatibility and intentional rejection. A migration test should inspect evidence and receipts, not simply count rows. Changes to a prompt or tool contract can also change the interpretation of a pending step even when the database schema stays identical. Record those versions and avoid silently combining old evidence with new assumptions in a remediation decision.
Permissions and retention
A recovery identity needs access to the relevant run and evidence, not unrestricted production administration. Separate read-only inspection from remediation dispatch, and preserve the original tenant boundary across a resumed run. Expired credentials should be reacquired through an authorized identity flow; storing a token inside a checkpoint is not a durable authorization strategy. A run surviving longer than a credential is normal.
Retain enough structured evidence to explain completion without keeping every sensitive raw payload forever. A useful report record can include source references, observed time and policy version. Deleting underlying evidence may make later re-evaluation impossible; expose that limitation instead of manufacturing a fresh source and calling it the original. Retention rules must cover backups and derived summaries as well as the active checkpoint table.
Observe progress and bound costs
Measure time in each step, failed transitions, repeated provider calls, incompatible-version counts and oldest unfinished run. Track completion by authoritative receipt rather than a final assistant sentence. A rising checkpoint age with healthy process uptime is a meaningful failure signal: the runtime is alive but the business workflow is not moving. Alert on that distinction before users repeatedly submit the same incident.
Set separate limits for provider attempts, elapsed run age and concurrent active runs. A model call may have a token cap while the surrounding workflow still retries indefinitely. Cost per completed report should include unsuccessful attempts and abandoned runs. The offline module has no provider charges; its call counter is a teaching proxy for work, not a monetary estimate. Choose actual cost thresholds using measured adapter behavior and the value of the incident class.
Acceptance is evidence of bounded recovery
Require successful restart from every persisted step, atomic local receipt behavior, visible incompatibility, rejection of malformed provider results and bounded handling of uncertain external effects before enabling automatic continuation. The original workflow establishes the first four within its narrow local scope. It does not include remote actions, leases, distributed concurrency or a hosted scheduler, so those need their own tests.
Use a shadow trial to compare recovered reports with uninterrupted reports on the same synthetic incidents, then repeat on authorized representative data. Evaluate diagnosis quality through independent checks; a perfectly recovered workflow can faithfully preserve a bad conclusion. The operational goal is understandable progress with controlled failure behavior. Prefer an explicit unresolved state that an operator can investigate over an apparently completed run whose final effect cannot be demonstrated.