LLM Gateway Architecture

A gateway is infrastructure; a router is application logic

Once an organization has more than one team calling LLM APIs, a pattern repeats: each team hardcodes its own API keys, its own retry logic, its own fallback-to-a-different-model logic, and its own cost tracking -- because there was no shared layer to put it in. An LLM gateway is that shared layer: a single internal service every application calls instead of calling providers directly, responsible for the concerns that are genuinely the same across every caller (credentials, rate limits, provider outages, cost visibility) so that no individual application has to re-solve them.

This is a different concern than agent request routing, which decides which model or agent should handle a given task based on the task's difficulty -- that's application-layer judgment specific to one product. A gateway sits underneath all of that: every app's router, including the difficulty-based one, still calls out through the gateway to actually reach a model. Confusing the two layers is the most common design mistake in this space -- teams build gateway-shaped logic (fallback, retries, rate limiting) separately inside each application's router instead of once, underneath all of them.

What a gateway is actually responsible for

Credential and rate-limit pooling. Individual teams calling a provider directly each hit their own per-key rate limit, even though the organization's aggregate usage is well under what the provider would grant a single, larger pooled key. A gateway holds the provider credentials centrally and pools quota across all internal callers, so one team's traffic spike doesn't need its own limit-increase request and another team's quiet period doesn't go to waste.

Request routing by cost, latency, and capability. The gateway is where "use the cheap model for this, the frontier model for that" gets enforced as policy rather than left to each application's discretion -- a routing table keyed on a request tag (which an application sets: tier=cheap, tier=quality, or a specific capability requirement like function calling or a long context window) maps to a concrete provider and model, so the mapping can be changed centrally (a cheaper model gets promoted to the default cheap tier) without redeploying every application that uses it.

Provider-outage fallback. When a provider has a degraded period, individual applications calling it directly all fail simultaneously with no coordinated response. A gateway can detect a rising error rate or latency spike from one provider and automatically fail over new requests to a configured secondary provider/model, transparently to the calling application -- the tier abstraction from routing is what makes this possible: "give me a quality-tier model" can be satisfied by more than one provider.

Cost attribution and budgets

Without a gateway, "how much did team X spend on LLM calls last month" requires reconciling provider billing dashboards against application logs after the fact, if it's answerable at all. A gateway sees every request, so it can tag and aggregate cost per caller (team, application, even per end-user if the calling application passes an identifier through) in real time rather than at month-end reconciliation.

This enables the thing cost visibility alone doesn't: enforceable budgets. A per-team monthly token budget, enforced at the gateway, degrades gracefully -- route to a cheaper model once 80% of budget is consumed, hard-stop or require an override once 100% is hit -- rather than the alternative of an unexpected bill arriving after the spending already happened.

request -> gateway
             |-- resolve caller identity + budget status
             |-- resolve tier -> provider/model (routing table)
             |-- check provider health -> fallback if degraded
             |-- check response cache -> return cached if hit
             |-- forward to provider, record cost + latency
             |-- return response, update budget counters

Response caching

A meaningful share of LLM traffic in most applications is exact or near-exact repeats -- the same support question, the same classification input, the same boilerplate generation request. A gateway is the natural place to cache responses, because it's the one point every request already passes through regardless of which application originated it, so a cache hit from one application can serve a repeat request from a different one.

Exact-match caching (hash the request parameters, cache the response, serve identical requests instantly) is straightforward and safe. Semantic caching (serve a cached response for a request that's merely similar, not identical, based on embedding distance) is a real cost lever at scale but introduces correctness risk the exact-match case doesn't have -- the general caching trade-offs (TTL choice, invalidation, stampede risk) are covered in caching architecture; the LLM-specific addition is that a "wrong" cache hit here doesn't error, it silently returns a plausible-sounding wrong answer, so semantic-cache similarity thresholds need to be tuned conservatively and monitored for drift, not just set once and left alone.

Centralized observability

Because every request already flows through one service, a gateway is the cheapest place to get organization-wide observability: latency and error-rate dashboards per provider and per model, cost trends per team, and a single point to trace an individual request's full round trip. Building the equivalent visibility with every application instrumenting its own provider calls separately means reconciling N different logging formats after the fact -- the gateway gets this for free, as a side effect of being the one chokepoint.

This observability layer is distinct from, and usually feeds into, agent-specific tracing (spans for a multi-step agent's plan/execute/reflect loop) -- see observability and tracing for agentic systems for that layer. The gateway's view is per-request (one LLM call in, one response out, with cost and latency); an agent trace is per-run (many gateway calls stitched into one logical unit of work). A mature setup has both, with the agent trace's spans linking out to the corresponding gateway request records.

When a gateway is overkill

A single team, single application, single provider setup gets little from a gateway beyond an extra network hop -- the coordination problems a gateway solves (pooled quota across many callers, cross-team cost attribution, organization-wide provider fallback) don't exist yet when there's only one caller. The pattern earns its complexity at the point a second team starts calling LLM APIs and the organization needs to decide, even informally, whether to coordinate; building the gateway before that point is solving a problem nobody has yet.

The migration path is usually incremental rather than a rewrite: stand up the gateway, point new applications at it directly, and migrate existing direct-to-provider applications one at a time behind a compatible interface, rather than requiring a coordinated cutover of every caller at once.

Failure modes specific to this layer

The gateway becomes a single point of failure for every application at once. Centralizing every LLM call through one service means a gateway outage takes down every application that depends on it simultaneously, which is a much larger blast radius than any one provider's outage would have been before the gateway existed. This is manageable -- run the gateway itself with the redundancy the rest of your critical infrastructure gets -- but it's a real trade-off against the coordination benefits, not a free win, and it's the argument teams reach for when resisting a gateway; the right response is to build the gateway to the reliability bar of critical infrastructure, not to skip building it.

Routing table drift from what applications actually need. A tier system ("cheap" vs "quality") is a simplification, and it degrades when an application's real requirement is more specific than a tier captures -- it needs a 200k-token context window, or function calling with parallel tool calls, or a specific model's particular behavior on structured output. A gateway that only exposes coarse tiers forces applications to route around it for anything that doesn't fit the categories, quietly recreating the direct-to-provider sprawl the gateway was meant to eliminate. The fix is to let routing keys express capability requirements explicitly (context length, tool-calling support, structured-output mode), not just a cost/quality tier, so the gateway can satisfy a real requirement rather than a rough proxy for one.

Caching correctness under prompt-template changes. An application that changes its prompt template but keeps the same cache key scheme (e.g. keying only on user input, not the full assembled prompt) will silently serve stale-template responses from the cache after deploying a prompt change. Cache keys need to incorporate everything that actually varies the request -- the full request payload, or a versioned template identifier -- not just the part of the input that looks user-facing.