Running an Agent Step Inside an Existing Workflow Engine: Callback Tokens, Heartbeats, Timeouts and Late Completions

Agent-era architecture · Practical technical guide · 26 October 2026

How to run a long agent step inside an existing workflow as a callback task: heartbeats a stalled model call cannot block, timeouts that close an attempt, retries with new tokens, rejected late completions, side effects keyed to the execution, and when a plain task is enough.

Where an agent fits in a workflow you already run

Many organisations already run their business processes in a workflow engine: AWS Step Functions, Temporal, Camunda, or an in-house state machine. When an agent arrives, the tempting move is to let it take over the whole process. The safer move is to make the agent one step of the existing workflow, so the engine keeps owning the order of steps, retries, timeouts and the audit trail, and the agent only does the part that needs judgement.

An agent run takes minutes, not milliseconds, so it does not fit a synchronous task. Workflow engines solve long external work with a callback: the engine generates a token, hands it to an external worker, and pauses the task until something sends the token back with a result. In Step Functions this is the wait-for-callback pattern, written as a resource ending in .waitForTaskToken, with the token passed as $$.Task.Token; the worker finishes with SendTaskSuccess or SendTaskFailure. Temporal offers the same shape as asynchronous activity completion, where the activity returns without completing and an external client completes it later. The rest of this guide uses Step Functions names, but the failure modes are the same in both.

What stays deterministic

The workflow keeps every decision about time and retries; the agent decides only what the result should be.

Concern	Owner	Deterministic?
What the step produces	Agent run	No; this is the judgement
When an attempt is considered dead	HeartbeatSeconds and TimeoutSeconds on the task state	Yes
Whether to retry and after how long	Retry policy on States.Timeout	Yes
Which token is valid	Workflow engine	Yes
Whether a side effect already happened	Idempotency key at the downstream service	Yes
What happens after the step	Workflow definition	Yes

The timing contract you are signing up for

Three settings govern the step. TimeoutSeconds is the maximum life of an attempt; the Step Functions documentation is explicit that heartbeats do not extend it. HeartbeatSeconds is the longest gap allowed between heartbeats, must be smaller than TimeoutSeconds, and closes the attempt with States.Timeout when exceeded. Both default to 99,999,999 seconds, and a task waiting for a token is otherwise held until the execution reaches the one-year service quota, so leaving them unset turns a crashed worker into an execution that waits for a year.

Two consequences matter for agents. First, when an attempt times out and the retry policy runs, the retry gets a new token; the documentation notes that a new random token is generated. Second, the old worker is not stopped. It finds out only when it next calls the service with its old token: SendTaskHeartbeat and SendTaskSuccess both answer TaskTimedOut when the token has expired or its task has already closed.

A worked example: a refund decided by an agent

A claims workflow sends each disputed charge to an agent worker through a queue with the task token. The agent reads the account history, decides, issues a refund through the payments API after about 200 seconds of work, and reports success at about 240 seconds. The worker sends a heartbeat every 30 seconds between agent steps; the task state has HeartbeatSeconds 120, TimeoutSeconds 900 and two retries.

On one run a model call stalls for 180 seconds after 60 seconds of work. Because heartbeats are sent between agent steps, none are sent during the stall, and the attempt is closed at 180 seconds. The retry starts at 185 seconds with a new token. The original worker resumes, issues its refund at 380 seconds and calls SendTaskSuccess at 420 seconds, which is rejected. The second worker issues its own refund at 385 seconds and succeeds at 425 seconds. The workflow history looks clean, one timeout and one success, while the customer has two refunds. The companion lab reproduces exactly this run and lets you change each setting.

Illustrative Python using boto3 Step Functions client method names; agent and payment interfaces are placeholders.

import json
import threading

def run_agent_step(sfn, token, claim, pay, agent):
    stop = threading.Event()

    def beat():                               # separate thread: a stalled model
        while not stop.wait(30):              # call cannot silence it
            try:
                sfn.send_task_heartbeat(taskToken=token)
            except sfn.exceptions.TaskTimedOut:
                stop.set()                    # our attempt is dead: stop working

    threading.Thread(target=beat, daemon=True).start()
    try:
        decision = agent.decide(claim, should_stop=stop.is_set)
        if stop.is_set():
            return                            # never act on a dead token
        key = claim["execution_id"] + ":IssueRefund"  # same on every attempt
        result = pay.refund(claim["charge_id"], decision["amount"],
                            idempotency_key=key)
        sfn.send_task_success(taskToken=token,
                              output=json.dumps({"refund": result["id"]}))
    finally:
        stop.set()

Three fixes, each for a different failure

Heartbeat from something that keeps running. A timer thread or a sidecar keeps the heartbeat alive while the model is busy. Heartbeats prove the worker process is alive, not that the agent is making progress; for progress, have the agent report steps and alert on a run that heartbeats but never advances.
Stop on TaskTimedOut. When a heartbeat is answered with TaskTimedOut, the attempt is gone. Stop the agent and do not perform any further side effect. In the example the old worker would stop at its next heartbeat, before the 380-second refund.
Key side effects to the execution, not the token. A key built from the execution ID and step name is the same on every attempt, so the payment service returns the first refund to the second worker instead of making another. A key derived from the token changes on every retry and protects nothing.

Migration steps

Model the agent step as a callback task in the existing workflow, with the token sent through the queue or API the worker already reads.
Set HeartbeatSeconds well below TimeoutSeconds, and TimeoutSeconds to the longest run you are willing to wait for, not the default.
Move heartbeats out of the agent loop into a timer thread or sidecar, and stop the agent on a TaskTimedOut answer.
Give every side effect inside the agent run an idempotency key derived from the execution and step, and check that each downstream service honours it.
Retry on States.Timeout with backoff and a small maximum, and route the final failure to a person through the workflow rather than from inside the agent.
Record the attempt number and token suffix in the agent trace so late completions can be matched to the attempt that produced them.

Failure modes and verification

Test the step with a fake agent that stalls on command. Assert that the attempt closes at HeartbeatSeconds, that the old worker stops at its next heartbeat, that exactly one refund exists for the claim, and that a late SendTaskSuccess is rejected without failing the worker loudly. Watch for a worker that treats TaskTimedOut on SendTaskSuccess as a crash and retries it, a queue that redelivers the original message with the old token after a visibility timeout, and output that exceeds the engine's payload limit, which turns a correct decision into a failed completion. In production, alert on late completions per day: each one is an attempt whose work the workflow had already given up on.

When this is unnecessary

If the agent call finishes within the engine's synchronous task limits and has no side effects of its own, a plain task with a timeout and a retry is enough, and none of the token machinery is needed. If the whole process is one agent conversation with no other steps, a workflow engine adds little. The callback pattern earns its keep when the agent is one step among many, its run is long, and its actions touch systems that must not see them twice.

Sources and scope

Checked against these sources on 26 October 2026. Examples and figures are illustrative, not measurements.

Run a long agent step as a callback task, set HeartbeatSeconds well below TimeoutSeconds, send heartbeats from something a stalled model call cannot block, stop the worker when the workflow answers TaskTimedOut, and key every side effect to the execution rather than to the token, because each retry gets a new one.