Define the operating contract
An incident pipeline is healthy when valid observations become durable state, required investigations finish within the agreed deadline, and failures remain inspectable. Broker availability alone is not that contract. A dashboard that shows successful ingestion can remain green while workers are stuck, evidence is stale or requests are being retried without progress. Start with a trace that follows one observation from receipt to an accepted investigation result.
Use three clocks: ingestion time, processing start and business completion. Store correlation identifiers rather than relying on log text to reconstruct joins. An observation identifier follows a delivery, an incident identifier joins related facts, and an investigation identifier names work. Keep these roles visible in logs and metrics. Reusing a single identifier for everything makes deduplication and latency analysis ambiguous once one incident creates several investigations.
Architecture and request sequence
Read the architecture as text
- Ingress metrics: Unique events and conflicts
- Queue age: Backpressure and expiry
- Relay attempts: Timeouts and retries
- Receipt ledger: Business completion evidence
- Operator: Pause, inspect and redrive
- Ingress metrics → Queue age: accepted work
- Queue age → Relay attempts: bounded batch
- Relay attempts → Receipt ledger: completion
- Relay attempts → Operator: exhausted
- Operator → Queue age: controlled redrive
Read the sequence as text
- Monitor → Operator: Report growing queue age
- Operator → Relay: Pause new attempts
- Relay → Store: Preserve pending identities
- Operator → Store: Inspect failed request
- Operator → Relay: Redrive approved batch
Exercise the dangerous delivery window
The reference tests inject a failure after the receiver accepts a request but before the sender marks it delivered. In a deployed system, this can result from a lost acknowledgment or a process termination. The correct response is not to assume failure and invent a new request. Retry the same logical operation with the same identity and consult the receiver's receipt when possible.
Extend the exercise to a durable receiver before permitting writes. Stop both processes, reopen their stores, and replay the pending request. Verify one business effect and an explainable final receipt. Also test the conflicting-content case: the same identifier with a new target must fail. A system that suppresses every repeated identifier without checking intent can conceal a serious routing or serialization bug.
Backpressure and poison messages
The reference drain loop is synchronous and unbounded over the pending snapshot; it is intentionally not a production scheduler. Add maximum batch size, per-attempt timeout, retry classification and a maximum retry age in an adapter implementation. A malformed permanent request should not occupy the worker forever, while a short outage should not turn every message into a permanent incident. Keep retry decisions explicit and observable.
A dead-letter record should retain a safe payload reference, failure category, attempt history and schema version. Redrive is a new operational decision that reuses the original business identity. It should not bypass validation or clear deduplication records. During a burst, prioritize current critical incidents and coalesce redundant observations where domain rules allow it. Report dropped, expired and superseded work separately so apparent throughput does not hide a shrinking coverage rate.
Permissions and evidence handling
Authenticate producers before accepting their claimed tenant or severity. Restrict workers to the incident scope they are assigned and validate callback ownership before updating results. A worker receiving an event is not automatically authorized to restart every service named in its payload. Treat incident descriptions, logs and retrieved documents as untrusted data even when they arrive through a legitimate internal integration.
Event payloads should usually carry evidence references instead of full sensitive logs. Apply access control when dereferencing them, set retention policies and redact secrets before model input. The local example contains synthetic data and no authentication layer. Its strict schema protects shape, not provenance. A production test should try a valid-looking cross-tenant event and confirm rejection before the worker adapter runs; another should prove that a poisoned log cannot change tool permissions.
Measure cost without rewarding retries
Count accepted unique observations, duplicates, conflicts, pending age, delivery attempts, exhausted requests and completed investigations. Cost per successful investigation should include failed and repeated model calls, not only the final successful attempt. Track token or provider spend in the real adapter alongside queue age so an incident storm does not silently create an uncontrolled bill. The offline module has zero provider spend and cannot predict a cloud bill.
Assume a worker budget of fifty investigations per minute for an initial trial, then measure whether the incident mix fits. That number is a proposed capacity limit, not a benchmark. If the queue exceeds the budget, choose an explicit degradation mode: combine related observations, defer low priority work, or ask an operator to triage. Spawning additional autonomous workers without shared rate limits simply moves overload to the model endpoint or downstream tools.
Acceptance and rollout
Accept the pattern when restart tests preserve state, duplicate delivery does not duplicate the receiver's effect, identifier conflicts are surfaced, and an exhausted request remains recoverable through a controlled path. Include an ordering test that demonstrates why maximum severity cannot represent recovery. Read evaluation foundations before treating these deterministic checks as evidence of investigation quality; a model's diagnosis needs separate grounded evaluation.
Roll out first as a read-only shadow subscriber, then enable reviewed investigation results for a small incident class. Keep a switch that pauses new work while preserving queued data and receipts. The original EventHub module is a compact failure laboratory, not a complete hosted platform. Use its tests to preserve the contracts when replacing SQLite or FakeWorker, and document which new failure modes the deployed components introduce.