Cost and Latency Budgets for LLM Products

Budget before you build, not after you measure

The common failure pattern: a team builds an LLM product, ships it, and only then discovers it costs four times the acceptable per-request budget or takes six seconds when the product needed sub-two. Both numbers were knowable at design time -- they're the sum of components whose individual costs and latencies are estimable before a line of code exists. Treating cost and latency as a budget to decompose and allocate upfront, the same discipline applied to any other system with a hard SLA, catches the mismatch while it's still a design change instead of a post-launch scramble.

Decomposing end-to-end latency into a budget

Start from the target -- say, a product requirement that a response starts streaming within 2 seconds -- and allocate that total across every component the request actually passes through before the model produces its first token: retrieval (if the request involves RAG, see RAG pipeline design for the retrieve/re-rank latency breakdown), any tool calls that must complete before generation can start, the gateway hop (see LLM gateway architecture), and the model's own time-to-first-token, which itself varies by model size and current provider load.

A concrete allocation for a 2-second budget: 100ms network/gateway overhead, 300ms retrieval, 100ms re-ranking, 1200ms model time-to-first-token, leaving roughly 300ms of margin. The purpose of writing the allocation down explicitly, per component, before building is that it turns "the product feels slow" from a vague post-launch complaint into a specific, attributable question: which component blew its allocated budget, and by how much -- the same diagnostic value a per-span latency budget gives you once you have tracing in place to actually measure each component against its allocation.

Decomposing cost-per-request into a token budget

The same exercise, in tokens and dollars rather than milliseconds. For a RAG-backed request: input tokens are dominated by retrieved context (say, 5 chunks at 400 tokens each, 2000 tokens) plus the system prompt and conversation history (call it 800 tokens), and output tokens are whatever the response actually needs (a support answer might target 150-300 tokens). At current frontier-model pricing in the few-dollars-per-million-tokens range for input and a few times that for output, a single request like this lands in the range of a fraction of a cent to a few cents, and the point of writing out the token breakdown is that it shows exactly which lever moves the number most: for most RAG-backed requests, retrieved context dominates input tokens by a wide margin, which means chunk count and chunk size (the same knobs from RAG pipeline design) are the primary cost lever, not prompt engineering elsewhere in the request.

Multiply the per-request cost by expected volume before committing to an architecture, not after -- a request that costs $0.02 looks trivial in isolation and becomes a real budget line at a million requests a day, and the point where that math needs to happen is during design, when switching to a cheaper retrieval strategy or a smaller model is a design decision, not a post-launch emergency migration.

The trade-off curve: fewer expensive steps versus more cheap ones

The central trade-off in both cost and latency budgeting is the same shape: one call to an expensive, capable model, or several calls to a cheaper, faster model with more orchestration around them. A single frontier-model call handling retrieval-augmented generation, response formatting, and a light safety check in one shot is simple and has one latency/cost line item; splitting that into a cheap classification call, a focused generation call, and a separate cheap safety-check call adds orchestration overhead (more network hops, more components that can fail independently) but can cost and cost less overall if each smaller call uses a model sized to its actual difficulty rather than paying frontier-model rates for a classification a much smaller model handles just as well.

There's no universally correct point on this curve -- it depends on how differentiated the sub-tasks' actual difficulty is. If every sub-task genuinely needs frontier-level capability, splitting adds orchestration cost for no savings. If the sub-tasks vary widely in difficulty (a classification step that a small model nails, next to a generation step that genuinely needs a capable model), splitting and routing each to an appropriately-sized model is close to free latency-wise (the steps can often run partly in parallel) and can cut cost substantially -- this is the same reasoning behind agent request routing, applied here specifically to the cost/latency budget rather than to routing policy in general.

Worked example: budgeting a support-chatbot request

Target: answer a customer support question, grounded in product documentation, in under 2.5 seconds end to end, at a cost the business can sustain at expected volume (100,000 requests/day).

At roughly 2270 tokens per request and typical mid-tier model pricing, this lands somewhere in the low-single-digit-cents range per request -- multiplied by 100,000 requests/day, that's a daily cost the business can evaluate directly against the value of automating those support interactions, which is the entire point of doing the arithmetic before building rather than after: the decision to proceed, use a cheaper model, or reduce retrieved-chunk count is a design choice made with real numbers, not a surprise on the first monthly invoice.

If the initial budget doesn't close -- say generation alone takes 3 seconds on the target model -- the levers to pull, in likely order of impact, are: reduce retrieved-chunk count (directly cuts input tokens and often generation time), switch to a faster model for generation (accept a capability trade-off), or split into a routed cheap/capable pair per the trade-off curve above rather than assuming the first model choice is fixed.