Design event-driven agents around facts, not conversations

Pattern 06 companion: Python example · Behavioral tests · Example license

The incident that arrives twice

An incident-response assistant receives an alert that a queue is growing. A monitoring integration retries after its acknowledgment times out, while a second monitor reports the same symptom. A conversational design might start three investigations and let the workers reconcile their messages later. Our proposed design instead distinguishes a delivery, an incident and an investigation request. Each has an identifier with a different lifetime. A repeated delivery can be discarded without pretending that two independent observations are identical.

The requirement is concrete: accepted observations must survive a worker restart; duplicate deliveries must not create duplicate local work; a more severe observation must not be erased by an older, milder observation. This example does not require a language model. The collaboration boundary is useful before a model proposes diagnoses, and stays useful when different models or deterministic analyzers become workers.

Architecture and request sequence

Component diagram for Design event-driven agents around facts, not conversations — Original reference architecture by Sandeep Belgavi. Arrows show the responsibilities and transitions described below.

Request sequence for Design event-driven agents around facts, not conversations — Illustrative sequence for this design; failure and recovery variants are explained in the article.

Contracts before the broker

The teaching envelope contains exactly four fields: id, incident, severity and version. The event identifier names one immutable observation, incident identifies the aggregate, severity is an integer from one through five, and version selects the schema. Reusing an event identifier with different content is a conflict, not a harmless retry. Unknown fields are rejected in this deliberately strict version so that accidental schema drift becomes visible during testing.

Separate events from commands in a larger system. An observation says that something was measured; an investigation request asks a worker to do something. A command needs an owner, expiry and completion contract. An event may have several subscribers. Avoid claiming that every subscribed agent has performed useful work merely because the broker accepted a message. Delivery, processing and business completion need separate receipts and separate measurements.

Three records in one local transaction

Our architecture places an inbox, incident table and outbox in the same SQLite database. Accepting an observation inserts its identifier, updates the aggregate and records downstream work atomically. The outbox relay later delivers that work through a provider adapter. This is a proposed educational architecture, not the documented internal implementation of a cloud event service.

The durable boundary matters more than the arrows. If the database transaction fails, none of those three changes should survive. If delivery succeeds but recording delivery fails, the relay may send again. Therefore the receiving worker also needs a stable delivery key. The reference worker deduplicates in memory only; a production receiver would need its own durable receipt. A local transaction cannot make a remote side effect atomic with local state.

Ordering is a domain choice

The example merges severity using a maximum operation. That decision is safe only for the narrow question, what is the highest severity observed for this incident? It intentionally cannot express recovery, a corrected false alarm or an operator downgrade. Those require separate event types, an explicit state machine and usually a version or causal ordering rule. Replacing this merge with the last message received would make network arrival order define business truth.

Partition work by incident when sequential decisions depend on earlier observations. Independent enrichment can still run concurrently, provided each result declares the input version it examined. A coordinator should reject or re-evaluate an obsolete enrichment result before it influences a remediation proposal. Collaboration becomes easier to reason about when workers exchange bounded evidence records instead of repeatedly appending unstructured messages to a shared conversation.

Capacity assumptions and tradeoffs

Assume, for planning only, twenty observations per second during a burst and one investigation worker completing two jobs per second. Without coalescing, that worker accumulates eighteen jobs per second. Adding a language model to every delivery makes the imbalance more expensive. An incident-level debounce window can combine related observations before investigation, but it introduces delay and must not suppress a newly critical severity. Measure the actual workload before choosing a window.

The SQLite reference is a single-machine teaching implementation, not a throughput benchmark. A hosted design could use a transactional database and durable broker with separate worker pools. More components buy independent scaling and isolation while adding failure modes, deployment work and operating cost. Start with the smallest design that preserves your required delivery and state boundaries; do not add agents merely to give every box a conversational identity.

What the reference proves

EventBridge documents retrying failed target deliveries and using a dead-letter queue for exhausted delivery attempts. That service behavior motivates asking what happens when delivery repeats; it does not establish exactly-once business processing. Our original example demonstrates local deduplication, atomic state changes and an explicit delivery adapter instead.

Use the evaluation foundation to define success before implementation: repeated input produces one inbox record, conflicting reuse fails, an injected transaction failure leaves no partial incident, and relay retries do not duplicate the fixture receiver's receipt. Those assertions are narrower and more useful than saying the agent collaborated successfully. The next article walks the actual module, including the gap between durable local state and the in-memory receiver.