Why it matters

Tail latency shapes user experience. Understanding shapes SLO.

Advertisement

The architecture

Long prompts: chunked prefill.

Long outputs: streaming.

Disaggregation for isolation.

Tail latency techniquesLong promptschunked prefillLong outputsstreamingIsolationdisaggregationMeasure p99, p99.9; long-tail hides in averages; user experience follows tail
Tail latency.
Advertisement

How it works end to end

Chunked prefill: bounded step.

Streaming TTFT priority.

Disaggregation isolates.

Cost-quality trade-off.