Why it matters

TTFT shapes UX. Understanding shapes latency SLOs.

Advertisement

The architecture

TTFT = queue + prefill + first decode.

Queue: admission delay.

Prefill: input tokens FLOPs.

TTFT breakdownQueue delaybatch waitPrefill timeinput FLOPsFirst decodesingle tokenPrefix cache reduces prefill dramatically; chunked prefill reduces per-step delay
TTFT.
Advertisement

How it works end to end

Queue: reduce via more capacity.

Prefill: chunked or cached.

First token: normal decode.