Timeouts for Streaming LLM Calls: First-Token, Idle and Total Budgets

A help-desk answer cut off at 60 seconds

The help-desk assistant at Belgavi's Book Store streams its answers. Geetha asks about a damaged box set on CASE-4103; the model thinks for 36 seconds before the first word, then writes for 47 more. The client has one rule: give up after 60 seconds. At 60 seconds it closes the connection, the half-written answer disappears and the retry starts from nothing. Meanwhile, on CASE-4108, the stream stopped producing text after 7.7 seconds and the same client sat there for another 52 seconds before noticing.

One number cannot express both "this answer is allowed to be long" and "silence is suspicious". A streamed call has three distinct phases that fail in different ways, and each deserves its own budget.

Three budgets instead of one

A first-token budget bounds the wait before the first content event: connection, queueing and any thinking the model does before it writes. A stall here looks like a request that never starts. An idle budget bounds the gap between content events once the answer is flowing; it catches a stream that started and then hung. An overall deadline bounds the whole call, so a stream that keeps trickling never runs forever; it should come from the caller, the way the Google SRE chapter on cascading failures recommends propagating one absolute deadline down an RPC tree rather than inventing one per hop.

The first-token budget is the one that varies most with the model. Reasoning models can spend tens of seconds before producing text. In the Claude streaming reference read on 2 November 2026, a thinking block configured with display: "omitted" streams no thinking text at all: it opens, receives one empty thinking_delta and a signature_delta, and closes. Decide whether your first-token budget counts any non-ping event or only the first text, and size it for the thinking effort you request.

What the SDKs actually enforce

The httpx2 documentation (the Pydantic-maintained continuation of httpx that both official Python SDKs now depend on) defines four timeouts: connect, read, write and pool, where read is the "maximum duration to wait for a chunk of data to be received". There is no total-request timeout in that list. A read timeout is therefore an idle timeout on bytes, reset by anything that arrives.

Both SDKs set the same default. In openai-python v3.42.1 and anthropic-sdk-python v2.3.0, read from their tagged sources on 2 November 2026, DEFAULT_TIMEOUT is an httpx2.Timeout of 600 seconds with a 5 second connect timeout, and httpx2's Timeout class assigns that first value to read, write and pool. The openai README describes this as "requests time out after 10 minutes"; for a stream it means up to ten minutes between chunks, with no cap on the whole stream. Both default to two retries, and the anthropic client refuses a non-streaming call whose max_tokens implies more than ten minutes, asking you to stream instead.

Two details change how a wrapper must work. First, the Anthropic streaming reference says a stream "might also include any number of ping events", with no interval documented, and anthropic-sdk-python v2.3.0 skips them in _streaming.py before your loop sees an event. A ping still resets the transport read timer, so a server that is alive but no longer generating can keep a read timeout satisfied indefinitely. Second, the retry loop in that SDK's _base_client.py returns as soon as the response status is a success; a stall or read timeout after the 200 arrives surfaces in your iteration and is not retried for you. The errors reference adds that an error such as overloaded_error can arrive as an event after a 200.

A wrapper that enforces all three

The wrapper below sits on any async iterator of events and recomputes, before each wait, whichever budget expires first. Only content events reset the idle clock; pings pass through without counting.

Runnable with Python 3.11 or newer, standard library only. The comments show its output on Windows; elapsed times depend on the platform timer, the outcomes do not.

import asyncio
import time


class StreamTimeout(Exception):
    pass


async def guarded(events, first_token, idle, deadline, is_content=lambda e: e != 'ping'):
    """Yield events; enforce budgets on content events only. Pings never reset a budget."""
    start = time.monotonic()
    last_content = None
    it = events.__aiter__()
    while True:
        now = time.monotonic()
        if last_content is None:
            budget, name = start + first_token - now, 'first-token'
        else:
            budget, name = last_content + idle - now, 'idle'
        if start + deadline - now < budget:
            budget, name = start + deadline - now, 'deadline'
        try:
            event = await asyncio.wait_for(it.__anext__(), timeout=max(budget, 0))
        except StopAsyncIteration:
            return
        except asyncio.TimeoutError:
            raise StreamTimeout(name) from None
        if is_content(event):
            last_content = time.monotonic()
        yield event


async def fake_stream(script):
    for delay, event in script:
        await asyncio.sleep(delay)
        yield event


async def run(name, script, **budgets):
    seen = {'content': 0, 'ping': 0}
    t0 = time.monotonic()
    try:
        async for e in guarded(fake_stream(script), **budgets):
            seen['ping' if e == 'ping' else 'content'] += 1
        outcome = 'completed'
    except StreamTimeout as err:
        outcome = 'timeout: %s budget' % err
    print('%-14s %-28s after %.1f s, %d content, %d pings'
          % (name, outcome, time.monotonic() - t0, seen['content'], seen['ping']))


budgets = dict(first_token=0.8, idle=0.3, deadline=3.0)
thinking = [(0.2, 'ping')] * 2 + [(0.1, 'text')] * 5
stalled = [(0.05, 'text')] * 3 + [(0.2, 'ping')] * 10
long_answer = [(0.05, 'text')] * 30
silent = [(5.0, 'text')]
asyncio.run(run('thinking', thinking, **budgets))
asyncio.run(run('pinging stall', stalled, **budgets))
asyncio.run(run('long answer', long_answer, **budgets))
asyncio.run(run('silent start', silent, **budgets))
# thinking       completed                    after 1.0 s, 5 content, 2 pings
# pinging stall  timeout: idle budget         after 0.5 s, 3 content, 1 pings
# long answer    completed                    after 1.9 s, 30 content, 0 pings
# silent start   timeout: first-token budget  after 0.8 s, 0 content, 0 pings

The thinking stream survives because 0.4 seconds of pings fit inside a 0.8 second first-token budget. The stall that keeps pinging is caught 0.3 seconds after its last text, which a read timeout would never do. The 30-chunk answer runs longer than both small budgets and completes, because only the 3-second deadline bounds total length. In real use, close the underlying response when a budget fires, or the connection keeps streaming into nothing.

What the lab shows

The companion lab simulates 200 synthetic streamed answers: 20 percent long (up to 110 seconds of generation), 10 percent with a 15 to 50 second thinking phase, 5 percent of attempts stalling, pings every 5 seconds. These are model figures from a seeded simulation, not measurements of any provider.

The total timeout trades one failure for the other: raising it from 60 to 120 seconds saves 32 healthy answers and doubles the wait on every stall. The per-read timeout never kills an answer while pings flow, but five stalled streams keep pinging and are never detected; the model assumes half of all stalls keep pinging, which no provider documents, and stops watching at 600 seconds, so 596.9 is a floor. Turn pings off and it catches every stall, but kills 14 thinking streams. The three budgets complete every healthy answer and detect every stall, with a p95 of 37.3 seconds dominated by stalls before the first token.

Retries and tradeoffs

• Retry only what a retry can fix. With the 60 second total timeout and one retry, the lab sends 47 retries and kills 68 healthy attempts instead of 36, discarding 4,080 seconds of streaming, because a retried long answer is just as long. Under three budgets the same retry rescues stalls: 199 of 200 complete.
• Tight budgets are false alarms. A 10 second first-token budget kills 18 thinking streams; a 5 second idle budget kills 12 answers with slow gaps; a 120 second deadline kills 4 long ones.
• Budgets per route. A reasoning model and a small classifier need different numbers; keep them next to the model name in configuration.
• Partial output. When a budget fires mid-answer, decide whether to show what arrived, resume from it or discard it.

Failure modes

• Treating the SDK default as a total. Ten minutes per chunk is not ten minutes per call.
• Counting pings as progress. Application budgets must be driven by content events, not bytes.
• One budget for every model. A first-token budget sized for a fast model fails every thinking request.
• Retries that ignore the deadline. A retry must fit in what remains of the caller deadline, not start a fresh one.

How to verify and measure

1. Record time to first content event, the largest inter-event gap and total duration per route, and set budgets above their observed p99 with margin.
2. Count budget firings by name (first-token, idle, deadline) and by model; a rising first-token count after a model change is a sizing problem, not an outage.
3. Inject stalls in a test double that keeps sending pings, and assert the idle budget fires.

Practice and related reading

Hands-on: Streaming LLM timeouts: one total timeout, a read timeout or three budgets, an interactive lab on synthetic data.

Further reading:

• SSE for LLM Streaming Responses, in depth
• LLM Streaming Output Architecture, in depth
• Pooled HTTP Connections That Die Mid-Request: Idle Timeout Mismatches
• Retry, Exponential Backoff and Jitter, in depth

Sources and scope

Checked against these sources on 2 November 2026. The wrapper output comes from running the snippet. SDK defaults were read from the tagged sources named above; the lab streams, their timing distributions and stall rates are synthetic.

• HTTPX2 documentation: Timeouts (connect, read, write and pool)
• openai-python v3.42.1: default timeout and retries in src/openai/_constants.py (README: Timeouts)
• anthropic-sdk-python v2.3.0: _constants.py, _streaming.py (ping handling) and _base_client.py (retries)
• Claude API documentation: Streaming messages (event flow, ping, error and thinking events)
• Claude API documentation: Errors (error types, errors after a 200, long requests)
• Google SRE book, chapter 22: Addressing Cascading Failures (deadlines, retries, graceful degradation)