Streaming LLM timeouts: one total timeout, a read timeout or three budgets

What this experiment demonstrates

The help-desk assistant at Belgavi's Book Store streams 200 synthetic answers, one per case from CASE-4101. By default 20 percent are long (up to 110 seconds of generation), 10 percent think silently for 15 to 50 seconds before the first word, 5 percent of attempts stall and never send more content, and the server sends a ping every 5 seconds of silence. A stalled server keeps pinging on half of the stalls.

A per-read timeout is reset by any byte, including a ping. The three budgets count only content events. A retry re-sends the same request: same length and thinking time, fresh gaps and stall. The simulation is seeded and deterministic; no model or network is called.

Run controlled comparisons

Start with the defaults: one 60 second total timeout. 153 of 200 answers complete, 36 healthy attempts are killed (most of the long answers and 8 thinking streams), all 11 stalls are detected but only at 60 seconds, a p95 wait of 56.9 seconds after the stall, and 2,160 seconds of healthy streaming are thrown away.

Now change one thing at a time. Raise the total to 120 seconds: 4 healthy answers killed, p95 wait 116.9 seconds. Switch to the per-read timeout of 30 seconds: no healthy answer killed, but 5 of 11 stalls are never detected because the stalled server keeps pinging, and the p95 wait is 596.9 seconds. Set the read timeout to 600, the shape of the SDK defaults: none of the 11 stalls is detected. Turn pings off: every stall is detected, but 14 thinking streams are killed; with a 10 second read timeout, 18. Switch to three budgets (first token 60, idle 15, deadline 300): 189 complete, none killed, all 11 stalls detected, p95 37.3 seconds; pings make no difference. Tighten each budget: first token 10 kills 18, idle 5 kills 12, deadline 120 kills 4. Turn on retries: under the 60 second total, 47 retries and 68 healthy attempts killed; under three budgets, 199 of 200 complete.

Inspect the trace and keep evidence

The table shows the first eight answers and then long, thinking, stalled and retried ones, with each attempt's outcome and which timer fired. The trace summarises short, long and thinking answers, one killed and one stalled example, and the time wasted. Start trace, Next step, Show all and the slider control how much of the trace is visible; the metrics always describe the whole run. Reset restores the defaults and Export JSON downloads the inputs, metrics and trace.

Limits and production translation

Real streams have heavier tails, network resets, error events after a 200 and providers that do not send pings. The stall model is binary and the run stops watching at 600 seconds. What carries over: a read timeout is an idle timeout on bytes, so pings can hide a stalled generator; a total timeout cannot be both long enough for long answers and short enough for stalls; budgets over content events separate the two; and a retry only helps failures it can change.