Why architecture matters here
Streaming agents fail on the boring parts. Client disconnects mid-response — is the work still running? Interrupt during a tool call — does the tool complete or abort? Client reconnects — can it resume without repeating the tool call? Slow client — does the server buffer forever?
The architecture matters because these are joint concerns of the runtime, protocol, and state management. Naive implementations lose work; over-engineered ones become impossible to debug.
With the pieces in mind, you can build a streaming agent that survives real network conditions.
The architecture: every piece explained
The top strip is the wire path. Client opens an SSE connection. Agent runtime runs an async loop backed by virtual threads or reactive streams. Model stream emits tokens token-by-token. Tool call events announce start and result — clients can render "agent is running search()".
The middle row is the resilience. Partial output deltas ship as they arrive. Interrupt handler handles user cancels and tears down downstream work (model call, tool invocation) with cancellation propagation. Resume from checkpoint uses a session ID to pick up after disconnects. Backpressure pauses the model stream when the client is slow, propagating pressure upstream.
The lower rows are state and ops. State store persists checkpoints per step so a resume can rehydrate the runtime. Observability tracks stream metrics — first-token latency, tokens/s, interrupt count — and trace across services. Ops handles timeouts, reconnect strategies, and graceful shutdown when the server rolls.
End-to-end flow
End-to-end: a user asks the agent a question. Client opens SSE. Runtime starts a model call; tokens stream to the client. Model calls a tool; runtime emits a tool_start event, runs the tool, emits tool_result. Model continues generating. User's network hiccup; SSE closes. Runtime observes the disconnect; because the user did not cancel, it continues to completion and checkpoints state. User's client reconnects with the same session ID; runtime replays the buffered output. User interrupts; runtime sends cancellation through the model call, aborts, and returns a final message. Metrics show p95 first-token 320 ms.