Agent step behind a callback token: heartbeats, timeouts, retries

What this experiment demonstrates

A workflow task state hands a callback token to an agent worker and then waits. The worker runs the agent for a fixed amount of work, issues one refund partway through, and reports success with the token. The task state closes an attempt when no heartbeat arrives within HeartbeatSeconds or when TimeoutSeconds passes, and a retry starts a new attempt with a new token. The worker of the closed attempt is not told; it keeps running until it next talks to the workflow.

The model follows the callback pattern of a workflow engine such as AWS Step Functions: a heartbeat resets the heartbeat clock but never extends TimeoutSeconds, and a token that belongs to a closed attempt is answered with TaskTimedOut. Times are seconds of wall clock from the start of the first attempt. Nothing calls a real workflow service or payment system.

Run controlled comparisons

Start with the defaults: 240 s of agent work, a refund after 200 s of work, heartbeats every 30 s sent between agent steps, HeartbeatSeconds 120, TimeoutSeconds 900, two retries after 5 s, and a 180 s stall after 60 s of work on the first attempt. The stall blocks the inline heartbeats, so attempt 1 is closed at 180 s. Its worker does not know: it issues the refund at 380 s and calls SendTaskSuccess at 420 s, which is rejected. Attempt 2 starts at 185 s, refunds again at 385 s with a key derived from its new token, and succeeds at 425 s. One order, two refunds.

Now change one thing at a time. Send heartbeats from a separate timer: the stall no longer silences them, attempt 1 finishes at 420 s and only one refund is made. Derive the refund key from the execution and state name instead of the token: the retry still happens, but its refund is a no-op. Make the worker stop when a heartbeat is answered with TaskTimedOut: the old worker stops at its 240 s heartbeat, before its refund, so there is one refund and no late completion. Set HeartbeatSeconds to 0: with only TimeoutSeconds 900 the stalled attempt is allowed to finish. Set the retries to 0: the task state fails at 180 s, but the abandoned worker still refunds.

Inspect the trace and keep evidence

The attempts table lists each attempt with its token, start, the time its work would finish, when it was closed, what side effect it produced and how its completion was answered. Red rows issued the refund after the workflow had already closed the attempt. The metrics summarise the run: the outcome and whether the workflow failed, attempts, timed-out attempts, late completions rejected, refunds issued, duplicate side effects, seconds of work done by abandoned workers, and when the workflow finished. Start trace, Next step, Show all and the slider control how much of the decision trace is visible; the table and metrics always describe the whole run. Reset restores the defaults and Export JSON downloads the inputs, metrics, attempts and trace.

Limits and production translation

The model treats heartbeats as exact, applies timeouts instantly and runs one worker per attempt; a real service has delivery delays, quotas and jitter, and a real worker may be restarted mid-run. The stall stands in for any long model or tool call that blocks the thread that would send heartbeats. What carries over: send heartbeats from something that keeps running while the agent waits, make the worker stop when the workflow says its token is dead, derive idempotency keys for side effects from the execution rather than from the token, and set HeartbeatSeconds so a dead worker is noticed long before TimeoutSeconds.