GPU Inference Latency — TTFT vs TPOT

Why it matters

Understanding latency components shapes serving optimization.

The architecture

Prefill: process input tokens. GEMM-heavy.

Decode: generate one token at a time. GEMV-heavy, memory-bandwidth bound.

How it works end to end

Prefill parallelizes tokens; benefits from larger batches.

Decode: 1 token at a time per sequence; batching helps throughput but not latency.

Optimize TTFT: prefix caching, chunked prefill.

Optimize TPOT: speculative decoding, faster hardware.