Why it matters

Streaming shapes UX. Understanding shapes design.

Advertisement

The architecture

Engine emits token-by-token.

Transport streams to client.

Client incrementally renders.

Streaming serve flowEnginetoken by tokenTransportSSE / WS / gRPCClient renderincrementalvLLM + TGI + TRT-LLM native streaming; OpenAI-compatible SSE common
Streaming serve.
Advertisement

How it works end to end

OpenAI-compatible SSE.

Backpressure handling.

Reconnect logic.

Token-level metrics.