Putting an Agent Step Inside an Existing Event Consumer: Per-Key Ordering, Model Rate Limits, Poison Events and Effects That Survive Redelivery

The consumer you already have

Most organisations that want an agent in a business process already move that process through events. An order service publishes OrderPlaced, a billing service consumes it, a dispute service listens for ChargeDisputed. The tempting change is to replace one deterministic handler with an agent: instead of a rules table deciding whether a dispute is refunded, escalated or rejected, a model reads the event, looks at the account and decides.

The consumer around that handler encodes guarantees that the old code got for free. Events for one key arrive in order because the key picks the partition. A handler that runs in two milliseconds never exhausts the poll interval. A handler that throws on bad input fails the same way every time, so the team built a dead-letter path long ago. A model call breaks all three assumptions at once: it takes seconds, it is rate limited by someone else, and it can fail on one specific input for ever while succeeding on its neighbours. This article keeps the consumer and changes what happens inside it.

What stays deterministic

Draw the boundary before writing code. The agent proposes a decision for one event; everything that touches the log or the database stays deterministic and testable without a model.

The agent never commits offsets, never chooses whether to skip an event and never writes the effect directly. It returns a structured decision, and the consumer either applies it through the same validated path the old handler used, or parks the event.

Ordering: per key, not per partition

A partition gives total order, which is stronger than most businesses need. What matters is that events for one account are applied in order: a refund must not be decided before the charge it refunds. Processing a partition strictly one event at a time preserves that, but with a four-second model call it turns a partition into a queue that drains at a quarter of an event per second.

The usual compromise is parallel across keys and serial within a key: up to N calls in flight per partition, never two for the same key, and never a later event of a key while an earlier one is unfinished. Committing then needs care, because events finish out of order. Commit only up to the first unfinished offset; a crash re-delivers everything after it, including events that already finished. That is why effects need idempotency, covered below. Kafka itself documents the other trap: if the poll loop stops calling poll() for longer than max.poll.interval.ms (five minutes by default), the consumer leaves the group and its partitions move to another member, so a handler that blocks on a slow model call can cause the very redelivery it is trying to avoid. Move model calls off the poll thread and keep polling.

Rate limits are backpressure, not errors

A 429 from the model provider means the shared budget for this window is used. Three reactions are common, and only one of them is right for an ordered consumer.

• Retry immediately. The request is repeated on the next loop, receives another 429, and burns provider-side rate-limit budget and logs. In the worked example below it produced 169 429 responses for the same work that needed 28.
• Wait for Retry-After. RFC 6585 defines 429 and allows a Retry-After header, whose meaning RFC 9110 specifies. Pause the affected partition with pause(), keep polling so the consumer stays in the group, and resume() when the window opens. Order is preserved because nothing behind the paused event for that key starts.
• Divert to a retry topic. The event is republished and the partition moves on. That is fine for independent events and wrong for keyed ones: later events of the same key overtake the diverted one. The worked example shows two order violations from this choice alone.

Poison events

Some events will never produce a valid decision: a malformed payload, a case outside the agent's instructions, a reference to a record that no longer exists. Retrying them for ever blocks the key, and with strict partition order it blocks the whole partition. Park the event after a small number of attempts, publish it to a parking topic with the attempts and the validation errors, and alert a person.

Parking one event is not enough when order matters. The later events of the same key must be parked with it, because applying them would act on a state the parked event was supposed to change first. Other keys continue. When the person resolves the parked event, its key is replayed in order from the parking topic. This is the rule that the dead-letter queue of a stateless handler usually lacks.

Worked example

Three partitions carry eight events each; keys are accounts. Event P1#2 (key e) is poison. The model allows six calls per ten-second window and each call takes four seconds. The consumer runs up to three calls per partition, one per key, waits for Retry-After on 429, parks after three attempts and holds later events of a parked key. It crashes at 25 seconds with offsets auto-committed every five seconds, the default auto.commit.interval.ms, and restarts five seconds later.

Read the table as relative, not absolute: the fixture is small and the clock is coarse. Without a crash, 24 calls (21 events plus three attempts on the poison event) at six per window take four windows, so the run ends at 38 seconds; the window, not the model latency, sets the pace. The crash re-delivers 14 events, eight of which had already applied their effect; only the effect key stops them being applied twice. Fourteen and eight are properties of the commit interval: commit after every finished event and the same crash re-delivers three events, none of them finished. The interactive lab linked below runs this fixture with every control exposed.

The handler

The runnable example in agent-decision-lab implements this boundary with an in-memory log, a fake model on a simulated clock and nine tests for ordering, parking, rate limits and redelivery. Checking the effect key before the model call has a second benefit: a redelivered event costs no model call at all. Its core is short:

Core of examples/python/agent_event_consumer in agent-decision-lab (standard library only).

def handle(self, event: Event) -> Outcome:
    effect_id = f"{event.topic}/{event.partition}/{event.offset}"
    if (event.partition, event.key) in self.parked_keys:
        self.parking.append((event, "earlier event of this key is parked"))
        return Outcome.HELD
    if self.effects.contains(effect_id):        # redelivered: no model call, no effect
        return Outcome.DUPLICATE
    try:
        decision = self.validate(self.decide(event))  # the only model call
    except RateLimited as limited:
        return Outcome.retry(limited.retry_after)     # the caller pauses the partition
    except InvalidDecision as bad:
        self.attempts[effect_id] += 1
        if self.attempts[effect_id] >= self.max_attempts:
            self.parked_keys.add((event.partition, event.key))
            self.parking.append((event, str(bad)))
            return Outcome.PARKED
        return Outcome.retry(self.retry_delay)
    self.effects.apply_once(effect_id, decision)
    return Outcome.APPLIED

Migration steps

1. Wrap the existing deterministic handler behind the same interface the agent will use, and keep it as the fallback for every key.
2. Add effect keys derived from topic, partition and offset (or the event id) to the existing write path, before any agent is involved. Redelivery already happens today; this fixes it for both handlers.
3. Move processing off the poll thread, add per-key in-flight tracking and commit only the finished prefix.
4. Introduce the agent in shadow: it decides, the old handler acts, and disagreements are logged by key.
5. Switch one event type or one tenant to the agent path, with 429 handling by pause and resume and a parking topic with key holds.
6. Watch parked events per hour, 429s per window, consumer lag and redelivered events after deploys; widen the cut-over only when they are boring.

When an agent is unnecessary

If the old handler is a lookup table that the business can still maintain, keep it. If the decision is needed within the request latency of a user, an asynchronous consumer is the wrong place for it anyway. If most events are routine and a few are ambiguous, let deterministic rules handle the routine ones and send only the ambiguous ones to the agent path: the rate limit, the cost and the parking volume all shrink with the share of events that need judgement.

Failure modes

• Model calls on the poll thread, which exceed max.poll.interval.ms under slow responses and trigger the rebalance that re-delivers everything.
• Committing the highest finished offset instead of the contiguous prefix, which silently skips unfinished events after a crash.
• A retry topic for keyed events, which reorders them.
• A dead-letter queue that parks one event but lets later events of the same key through.
• Effects written by the agent directly, without a key, so redelivery applies them twice.
• Treating 429 as an error and retrying immediately, which multiplies calls without adding throughput.

Practice and related reading

Hands-on: Agent event consumer: rate limits, poison events and redelivery, an interactive lab on synthetic data.

Further reading:

• Kafka consumers: positions, commits and the group coordinator
• Retry and backoff patterns
• Making an existing API safe for agent callers

Sources and scope

Checked against these sources on 21 October 2026. Examples and figures are illustrative, not measurements.

• Apache Kafka 4.2 Javadoc: KafkaConsumer (liveness, max.poll.interval.ms, pause and resume)
• Apache Kafka 4.2 documentation: consumer configs (auto.commit.interval.ms, max.poll.interval.ms, max.poll.records)
• RFC 6585: Additional HTTP Status Codes (429 Too Many Requests)
• RFC 9110: HTTP Semantics (Retry-After)