Why it matters

ITL shapes streaming feel. Understanding shapes SLOs.

Advertisement

The architecture

ITL = decode step time.

Decode step: read weights + KV + compute.

ITL breakdownDecode stepone tokenBandwidth-boundweight + KV readSpeculativeK tokens per stepUsers perceive slow streaming above ~50ms/token; below ~30ms feels instant
ITL.
Advertisement

How it works end to end

ITL_target: often 30-50ms/token.

Speculative decoding effectively lowers by acceptance rate.

Small batch: better ITL.