Why it matters

Prefill math shapes TTFT. Understanding shapes latency.

Advertisement

The architecture

FLOPs = 2 * N * L (params * tokens).

Time = FLOPs / effective GPU TFLOPs.

Batch parallelism.

Prefill mathPrefill FLOPs2 * N * LGPU TFLOPseffectiveTTFTFLOPs / TFLOPsChunked prefill splits into steps; overlap with decode of other requests
Prefill math.
Advertisement

How it works end to end

Llama 70B, 2k input: ~280 TFLOPs.

H100 bf16: 989 TFLOPs peak; ~500 sustained.

TTFT ~ 0.5s.