An AI gateway is the reverse proxy that every LLM call in an organisation passes through on its way to a model. Calling it a load balancer for models undersells it. A load balancer decides where a request goes; a gateway decides whether the request is allowed at all, on whose budget it lands, how many tokens it may burn, what happens when the backend returns a 500, what may be served from cache, and what is written to the audit log. It is a policy plane, and everything distinctive about it follows from one awkward fact: the resource being governed is priced in tokens, and the token count is not known until the response has finished streaming.
The gateway is a policy plane, not a router
Without a gateway, every application that talks to a model carries its own copy of the same concerns: a provider credential in an environment variable, a retry loop somebody wrote in a hurry, an ad-hoc timeout, no shared view of spend. Multiply by thirty teams and you have thirty answers to ‘who may spend how much on which model,’ none enforceable.
A gateway collapses those answers into one enforcement point. Because it sits on the request path, it is the only component that can see every call, price it, attribute it, and refuse it. That is the whole architectural argument: policy you cannot enforce is documentation. Which backend actually serves a request — the model-selection question — is a downstream concern the gateway delegates to a routing layer.
Identity: virtual keys instead of provider credentials
The first job is authentication, and the design that makes everything downstream possible is the virtual key. Callers never hold a real provider credential; they hold a key the gateway issued, which resolves to a record: owning team, cost centre, permitted models, the budget it draws on, the rate limits that apply, and whether its traffic may be logged in full.
The indirection buys three things that are painful to retrofit. Rotation becomes local — revoking one team’s key touches nobody else, and rotating the upstream credential is invisible to callers. Attribution becomes exact, because every request already carries the identity that will be billed rather than a header somebody hopefully set. And blast radius shrinks: a key leaked into a public repository spends against one capped budget, not an unlimited account. Issue keys per service and per environment, not per human.
Budgets: turning tokens into a spend ledger
Budget enforcement is where the gateway earns its keep and where the naive implementation fails. The obvious design — check the running total before forwarding, add the cost afterwards — has a hole in it. Cost is known only after the response completes, because output tokens are not knowable in advance. With a hundred requests in flight against a nearly exhausted budget, all hundred pass the pre-check and the cap is blown.
The fix is reservation. On admission the gateway reserves a pessimistic estimate — prompt tokens counted exactly, output tokens assumed at the request’s max_tokens — and decrements the budget by that amount. When the response finishes it reconciles against reported usage and refunds the difference. That bounds the overshoot to the concurrency level, a number you control. Layer the caps: per-key daily, per-team monthly, and an organisation-wide ceiling, rejecting if any layer is exhausted.
Rate limiting on tokens, not requests
Classical API gateways limit requests per second, because for a REST endpoint one request is roughly one unit of work. For an LLM that assumption collapses. A twenty-token classification and a hundred-thousand-token document summary are both one request, yet differ by orders of magnitude in cost and in serving capacity occupied. A requests-per-minute limit generous enough for the first is catastrophic for the second.
So the meaningful limit is tokens per minute, usually enforced alongside a much looser request limit that exists only to stop pathological loops. The mechanics are a token bucket keyed by virtual key, refilled at the allowance rate, debited by the same reserve-then-reconcile estimate the budget uses. Two details matter. Charge input and output tokens separately if they are priced differently, since summarisation and generation stress different limits. And when you reject, return a 429 with a Retry-After header, so well-behaved clients back off instead of hammering.
Timeouts, retries, and failover across providers
The gateway is the natural home for resilience because it is the only place that knows about more than one backend. The base layer is unglamorous: a connect timeout, a time-to-first-token timeout, and an overall deadline. Generation is slow enough that one total timeout is a blunt instrument — a request with no first token after ten seconds is stuck, while one still streaming at ninety seconds may be perfectly healthy.
Retries need discipline. Retry connection failures, 5xx responses, and 429s with exponential backoff and jitter; do not retry 4xx validation errors, which will fail identically forever. Cap attempts and charge retries to the caller’s spend, because a retry storm is a bill. Failover to a second backend adds one constraint: the request must be re-expressible against the alternate model, and quality is not identical, so tell the caller which model actually answered. Wrap each backend in a circuit breaker so a provider outage stops consuming timeouts for everyone.
Caching: exact match, and what must never be cached
The cheapest request is the one never forwarded. A gateway cache keys on a canonical form of the request — model identifier, full message list, and every parameter that affects output — and returns the stored response on a hit. It suits genuinely repetitive traffic: fixed system prompts over a small input set, evaluation harnesses replaying a suite, clients resending an identical body.
Three rules keep it safe. The cache key must include the tenant or key scope, or one team’s response leaks into another team’s reply. Any sampling temperature above zero means the cached answer is one draw from a distribution, so caching silently removes variation the caller may have relied on. And responses shaped by tool calls or retrieved documents go stale the moment the data changes, so they need short time-to-live values or none at all. Provider-side prompt caching is a different mechanism, operating on prefixes inside the serving stack, and not a substitute for this one.
Logging and redaction: the audit trail you will need
Every gateway is eventually asked ‘what exactly did we send that model in March?’ That means logging, and logging LLM traffic is uncomfortable, because prompts are the most sensitive payload in the system. They contain customer records, source code, credentials pasted by mistake, and whatever a user typed into a chat box.
Split the log into two streams with different rules. Metadata — timestamp, key, model, token counts, latency, status, cache hit, backend — is cheap, non-sensitive, and the basis of cost attribution, so retain it for every request. Bodies are expensive and hazardous: log them under a per-key policy, mask obvious secrets and identifiers before storage, sample rather than capture everything on high-volume paths, and give them short retention with real access controls. Redaction at write time is the only redaction that helps; a body written unredacted is already a disclosure.
Streaming pass-through constrains the whole design
Most LLM traffic streams, and that fact rules out an entire class of gateway implementation. A proxy that buffers the full response before applying policy destroys the property users care about most: tokens appearing as they are generated. The gateway must forward each chunk immediately while doing its own work on a tee of the stream.
The consequences ripple outward. Any policy needing the complete output — final token accounting, cache writes, output inspection — can only run after the last chunk, so the gateway holds per-request state for a connection that may last minutes. That forces an asynchronous, event-driven concurrency model; a thread-per-connection server exhausts itself at trivial load. It also breaks retry: once the first byte has reached the client you cannot transparently fail over, because the client has already seen a partial answer. So retry freely before first token and stop after it — time-to-first-token is the moment the failover window closes.
The latency tax and how to keep it small
A gateway is an extra network hop, and honesty about its cost is what keeps teams from routing around it. The tax has three parts: the hop itself, including TLS termination and re-establishment toward the backend; the policy lookups for key, budget, and rate limit; and any serialisation of the request body.
Only the second is really under your control, and it is the one most often mishandled: three sequential round trips to a shared counter store before the request is even forwarded add up, on every call. Keep key records in an in-process cache with short expiry and invalidation on change. Collapse the rate-limit and budget checks into one round trip by scripting check-and-decrement server side instead of read-modify-write. Write ledger and log entries off the response path. And measure the tax as added time-to-first-token, not added total latency — against a multi-second generation the total is flattering and hides a regression users would feel.
Operating a tier-0 dependency
The moment the gateway is mandatory it becomes the most critical service in the AI stack: when it is down, no team can call any model. Deployments must therefore be rolling and connection-draining, because restarting a node kills in-flight streams, and configuration — keys, budgets, limits — should reload without a restart.
The sharpest question is what happens when the gateway’s own state store is unavailable and it cannot check a budget or a limit. Failing closed protects spend and takes down every AI feature you have; failing open keeps them running and risks an unbounded bill. Most teams split it: fail open on rate limiting, a fairness mechanism, and fail closed on hard budget caps, a financial control. Decide deliberately and write it down — the incident is not the moment to discover your gateway’s opinion on the matter.