An agent step that outlasts the queue visibility timeout is delivered again and again. Heartbeats, timeout sizing, idempotent effects and when to move the step out of the queue.
One help-desk case, five replies
The help desk at Belgavi's Book Store already had a queue. Each new case, such as CASE-4210 from a customer whose copy of a novel arrived with a torn cover, became a message on a standard queue. A worker received it, looked up the order, wrote a reply, sent it and deleted the message. The step took about two seconds, the queue had the default visibility timeout of 30 seconds, and nobody thought about either number for years.
Then the team put an agent inside the worker. The step now calls a model to read the case, calls two tools, calls the model again to draft the reply and only then sends it. On a normal day that takes 15 to 25 seconds. On a busy afternoon some steps took 45. The customer on CASE-4210 received five nearly identical replies over two and a half minutes, and the message ended up in the dead-letter queue, where the on-call engineer found it the next morning, marked as a failure even though every attempt had succeeded.
Nothing in the queue misbehaved. The queue did exactly what its contract says. The contract was written for steps that finish well inside the timeout, and an agent step is a different kind of work: long, variable and dominated by remote calls whose latency the worker does not control. This guide walks through the mechanism, the arithmetic of the failure, the fixes that keep the queue and the fixes that move the long step elsewhere.
The contract the old worker relied on
A queue with a visibility timeout gives at-least-once delivery. When a consumer receives a message, the queue hides it from other consumers for the visibility timeout. If the consumer deletes the message before the timeout ends, the message is gone. If not, the message becomes visible again and the next receive gets it. Amazon SQS documents a default of 30 seconds for a queue, lets you change it per queue or per message, and caps the total at 12 hours from the first receive. It also states plainly that delivery is at least once, so a message can arrive twice even inside the timeout.
Three more details matter for the agent case. Every receive returns a new receipt handle, and SQS says you must use the most recently received handle to delete the message, otherwise the message might not be deleted. Every receive also increments the receive count. A redrive policy moves the message to a dead-letter queue once the receive count reaches maxReceiveCount. None of these rules know how long your step takes. They only know when you last received and whether you deleted.
The old worker was safe by accident: a two-second step against a 30-second timeout left a margin of fifteen times the work. Duplicate deliveries were rare and harmless because the step was quick and the reply service saw the same case twice at most.
Replaying the incident second by second
The companion model replays CASE-4210 with whole seconds, one message and a pool of consumers that is always polling. With a 30-second timeout and a 45-second agent step, the timeline is:
- t=0 s: consumer A receives the message with handle h1 and starts the step.
- t=30 s: the timeout expires, the message is visible again, and consumer B receives it with handle h2.
- t=45 s: A finishes, sends reply 1, and deletes with h1. That handle is no longer the latest, so the message stays.
- t=60 s: the timeout of B's receive expires and consumer C receives the message.
- t=75 s: B finishes and sends reply 2. Its delete uses a stale handle too.
- The pattern repeats until the fifth receive at t=120 s. At t=150 s the receive count has reached 5, so the queue moves the message to the dead-letter queue.
- t=165 s: E finishes, sends reply 5 and finds nothing to delete.
The model treats a delete with a stale handle as not removing the message, which is the cautious reading of "might not be deleted". The cost is not only five replies. The model reports 225 model-seconds spent for 45 seconds of useful work, and an on-call alert for a dead letter that was never a processing failure. Dead-letter queues are designed to quarantine messages that fail repeatedly. Here they quarantine messages that are merely slow, which corrupts the signal you use to find real failures.
Fix one: a heartbeat that extends visibility
The direct fix keeps the queue and makes the consumer say "still working" before the timeout runs out. In SQS that is ChangeMessageVisibility with the current receipt handle. The consumer starts a background timer when it receives the message and extends the visibility at a fixed interval while the step runs.
import threading
def process_with_heartbeat(sqs, queue_url, msg, step, interval=10, extend_to=30):
stop = threading.Event()
def beat():
while not stop.wait(interval):
sqs.change_message_visibility(
QueueUrl=queue_url,
ReceiptHandle=msg["ReceiptHandle"],
VisibilityTimeout=extend_to, # seconds from now
)
t = threading.Thread(target=beat, daemon=True)
t.start()
try:
result = step(msg) # model calls, tools, reply
finally:
stop.set()
t.join()
sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=msg["ReceiptHandle"])
return resultThe interval is the part teams get wrong. In the model, a heartbeat every 10 seconds against a 30-second timeout keeps ownership: the extensions at t=10, 20, 30 and 40 s push the deadline to t=70 s, the step finishes at t=45 s and the delete succeeds. A heartbeat every 30 seconds does nothing useful, because the first beat is scheduled for the same instant the timeout expires and the message has already been handed to consumer B. The rule of thumb is an interval no more than a third of the extension, so that one lost or delayed heartbeat call does not lose the message.
A heartbeat has costs. It is another call per message per interval, it must stop when the step ends or the message stays hidden after a crash until the last extension lapses, and it cannot extend past the 12-hour ceiling. It also hides genuine hangs: a step stuck forever keeps renewing forever unless the heartbeat thread also checks a step deadline.
Fix two: size the timeout from the latency you measure
The simplest change is a larger timeout, chosen from the step's measured latency rather than from the old worker's habit. The model shows that a 60-second timeout makes the 45-second step finish with one receive. The weakness is the tail: agent steps have long tails because a single slow model call or a retried tool can double the step. A timeout set at the median produces duplicates on the tail; a timeout set at a generous maximum delays recovery after a real crash by the same amount, because a crashed consumer's message stays hidden until the timeout runs out.
A workable compromise is a moderate timeout plus a heartbeat, with an explicit step deadline that the worker enforces itself. The timeout bounds recovery after a crash, the heartbeat covers healthy slow steps, and the deadline stops the worker from renewing a step that is stuck.
Fix three: make the effect idempotent, because duplicates still happen
Even with a perfect heartbeat, the queue promises at least once. The reply to the customer must therefore be safe to attempt twice. The companion model adds an idempotency key derived from the case and the step: the reply service records the key with the first reply and refuses the second. With the key on and no heartbeat, the customer gets one reply instead of five. The message still goes to the dead-letter queue after five receives and the model still spends 225 model-seconds, which is the point: idempotency removes the visible harm but not the waste or the false dead letter. You need both the key and correct visibility handling.
Choose the key from what the step means, not from transport details. The receipt handle changes on every receive and the model's call identifiers change on every attempt, so neither works. "CASE-4210, first-response reply" does.
Fix four: move the long step out of the receive loop
Sometimes the right answer is that a queue consumer should not hold a message for a minute at all. The pattern that ages best is to receive the message, record durable intent ("CASE-4210 needs a first response") in a store keyed by the case, delete the message within seconds, and let a durable workflow or a job table drive the agent step with its own retries, timers and status. SQS's own documentation points to Step Functions or smaller steps when processing exceeds its limits. The queue goes back to doing what it is good at: handing over work quickly. The agent step gets a home that understands long-running work.
This is also where the deterministic parts stay deterministic. Which case is due, how many attempts it has had, when to give up and page a person: these are decisions for the workflow store, not for the model.
When none of this is needed
If the agent step reliably finishes in a few seconds, for example a classification call with a small prompt and a strict output limit, the old contract still holds with a modest timeout increase. Measure first. Plot step duration percentiles and receive counts per message for a week. If the 99th percentile is well under a third of the timeout and the receive count histogram is all ones, leave the queue alone and spend your effort elsewhere.
Verifying the change
- Alert on the ApproximateReceiveCount of messages you delete. A healthy agent consumer deletes almost everything on the first receive.
- Track dead-letter arrivals together with the step duration of their last attempt. Dead letters with successful attempts are visibility problems, not failures.
- Log the receipt handle generation (first, second, third receive) on every side effect, so a duplicate reply can be traced to a redelivery.
- In a test environment, inject a slow model response that exceeds the timeout by a few seconds and check that exactly one reply is sent and the message is deleted.
- Kill a worker mid-step and check that the message reappears after the last extension lapses and is processed once.
Run the example yourself
The visibility-timeout model with receipt handles, heartbeats, maxReceiveCount redrive and an idempotency key, with tests for every timeline quoted here. Companion code on GitHub; its README lists the prerequisites, exact commands, expected output and limits.
Interactive experiment
Queue visibility timeout: when a slow agent step is delivered twice. Set a visibility timeout, an agent step time and a heartbeat, then watch redeliveries, duplicate replies and dead letters in a decision trace. Synthetic inputs; read its assumptions and limits before drawing conclusions.