Agent event consumer: rate limits, poison events and redelivery

What this experiment demonstrates

Three partitions of a dispute topic carry eight events each, and every event's key is an account. A consumer sends each event to an agent whose single call takes a few seconds and is limited to a number of calls per ten-second window; a call over the limit receives 429 with a Retry-After that points at the next window. Event P1#2 is poison: the agent never produces a valid decision for it. The consumer can crash at a chosen second and restart five seconds later from its committed offsets, which delivers again every event after the last commit.

The page advances a clock in whole seconds and records every decision: which event started, which got 429, which failed validation, which was parked, which applied its effect and which was recognised as a duplicate. Order is checked per key: an effect applied while an earlier event of the same key has not been applied counts as an order violation. Nothing calls a model.

Run controlled comparisons

Start with the defaults: six calls per window, four-second calls, parallel across keys, wait for Retry-After, park the poison event after three attempts and hold the rest of its key, crash at 25 seconds with offsets committed every five seconds, effect keys on. The run finishes at 58 seconds with 21 effects applied once each, three events parked, 28 responses of 429, and eight redelivered effects blocked by their keys. Turn the effect keys off and the same crash applies eight effects twice. Commit after every finished event instead and only three events are delivered again.

Now change one thing at a time. Retrying on the next tick instead of waiting multiplies the 429 count from 28 to 169 without finishing any sooner. A retry topic finishes earlier but applies two effects out of order, because later events of a key overtake the diverted one. Processing in parallel while ignoring keys gives one order violation. Letting later events of a parked key continue gives two. Retrying the poison event for ever keeps the run from finishing at all, and one event at a time per partition is slowest at 65 seconds. Set the crash to 0 to see the run without redelivery.

Inspect the trace and keep evidence

The event table lists all 24 events by partition and offset with their key, attempts and final state; parked events are highlighted, and anything not finished is marked. Metrics show the finish time, effects applied, model calls, 429 responses, parked events, order violations, duplicate effects, duplicates blocked and events delivered again. Start trace, Next step, Show all and the slider control how much of the decision trace is visible; repeated 429 responses are shown for the first few events only, and the metrics always describe the whole run. Reset restores the defaults and Export JSON downloads the inputs, the table and the trace.

Limits and production translation

The fixture is 24 events, one consumer and a fixed-window limiter on a one-second clock, so absolute times mean little; compare runs with each other. Real consumers poll in batches, rebalance when the poll loop stalls past max.poll.interval.ms, and share rate limits with other services. The crash loses in-flight calls without modelling their side effects, which is the right assumption only when effects are applied after validation, as here. What carries over is the structure: per-key order, 429 as backpressure on one partition, parking with key holds, commits of the finished prefix and effect keys on every write.