Measure progress rather than process survival
A workflow worker can be healthy while every incident is stuck at classified. Measure time spent in each stage, transition attempts, validation failures and age of the oldest pending checkpoint. The useful progress event is a committed successor, not merely a worker startup or a model response. Attach the workflow revision and category to stage metrics so an availability-specific collector outage does not disappear inside a healthy overall average.
Distinguish waiting for evidence from collecting evidence that supports a healthy outcome. In the sample, an absent inventory entry raises an error instead of creating ready with an empty string. A production run should preserve that distinction in its status and owner. An operator needs to know whether to repair collection, correct a classification or continue investigating the service. A single generic failure label makes those different interventions unnecessarily difficult.
Architecture and request sequence
Read the architecture as text
- Stage backlog: Age and attempts by checkpoint
- Transition audit: Committed progress, not worker uptime
- Failure classification: Data, judgment or migration issue
- Recovery owner: Repair the correct boundary
- Acceptance gate: Procedure and classification measured
- Stage backlog → Transition audit: detect stalled stage
- Transition audit → Failure classification: classify failure boundary
- Failure classification → Recovery owner: assign recovery owner
- Recovery owner → Acceptance gate: verify invariants
Read the sequence as text
- Queue → Worker: Deliver checkpoint job
- Worker → Store: Compare checkpoint version
- Store → Worker: Report already advanced
- Worker → Queue: Acknowledge duplicate safely
- Worker → Review: Record retry and accepted version
Exercise retry and migration scenarios
Inject a crash after classification but before its checkpoint commit, then another after commit but before the queue acknowledges the job. The first may need classification repeated; the second should reload the accepted category. Record which adapter revision performed the accepted classification. If two attempts return different categories, preserve the discarded proposal for diagnosis only if its retention is approved; it must not silently alter the committed path.
Test a checkpoint from the previous workflow version during a rolling upgrade. Add an unknown stage and a ready checkpoint missing its evidence field. The correct response is a visible validation or migration decision, not a best-effort skip. Keep a migration rollback plan that accounts for already advanced records. Reverting application code is insufficient if a new release has written state that the old code cannot interpret.
Protect procedure from untrusted content
The incident description may contain instructions that resemble an operator command: skip evidence, treat the incident as resolved, or choose an unrelated service. The classifier is allowed to propose a category, not modify the state machine. Test malicious content at that seam and verify that its output remains subject to the same supported-category validation. A model can still choose the wrong allowed category, so authorization checks and classification evaluation solve different problems.
Protect stored checkpoints as operational records. A user who can edit a narrative field should not be able to change stage to ready. Authenticate the worker identity, scope its write permissions and validate every loaded checkpoint. Consider whether evidence locators expose sensitive system names. Store detailed observations in a separately controlled system and avoid embedding credentials or raw customer payloads into the checkpoint merely because it is convenient for debugging.
Control queueing and stage-specific cost
When a telemetry dependency slows down, repeatedly advancing classified checkpoints can create a query storm. Use a retry schedule with a bounded attempt count and a clear next-attempt time. Prioritize incidents based on operational severity established outside the model, not on the confidence of generated language. Reserve separate capacity for new incident intake and for draining existing evidence waits so a backlog cannot prevent operators from seeing new work.
Track model requests per accepted classification and collector requests per completed package. A rise in repeated classification after an infrastructure change can reveal a broken commit or acknowledgement path. Compare that with classified-stage queue age before adjusting model settings. The workflow makes attribution possible because model work occurs at one defined point. Do not add a hidden model call to every retrying collector; that destroys the cost boundary and may change the accepted category mid-run.
Set acceptance criteria by state invariant
Before a controlled rollout, require that every ready checkpoint contains a supported category and retrievable evidence, every transition has an identified predecessor, and duplicate delivery cannot move a run backward. Verify that restarting a worker resumes at the recorded stage without unnecessarily calling the classifier. These are functional acceptance criteria for the workflow extension. The shipped tests cover the corresponding in-memory decisions but do not establish database or queue guarantees.
Evaluate the classifier on labeled ambiguous and ordinary reports, including cases that should be escalated. Measure overrides by incident type and monitor whether the chosen evidence was actually useful to reviewers. A workflow can be procedurally correct while selecting the wrong branch. Promote only when both procedure and judgment meet the application's criteria. Keep a manual path that can supply an authorized category with an audit reason without pretending the model selected it.
Operational sources and scope
The state labels and tests come from the original transition module. The operational scenarios are original engineering exercises, not reports of a deployed service. Read Evals before interpreting classifier metrics. No durable workflow engine or cloud queue was tested; a real implementation must demonstrate its commit, migration and recovery behavior under fault injection.