Why it matters
QPS math for LLM is nuanced. Understanding shapes capacity planning.
Advertisement
The architecture
QPS = throughput_tokens / avg_tokens_per_query.
Include prefill + decode.
Advertisement
How it works end to end
Query has input + output tokens.
Compute both.
Long tail dominates.