Evidence label: deterministic synthetic model. This page runs a small JavaScript model of a shared LLM API. Six synthetic tenants send requests with random prompt and output lengths; a gateway admits or rejects each one with per-tenant request or token buckets; and a backend of four replicas serves admitted requests in the chosen queue order, slowing every sequence as the batch grows. Nothing here calls a model or a real gateway; all numbers are calculated estimates that make the mechanism visible.
Play the lab
How to play
Choose the limit type, the per-tenant limit and the bucket size, then the queue order at the backend and the batch tenant's request rate. Every change replays the same two minutes of traffic with the same seed. Toggle the runaway agent and the interactive burst to stress your setup, and use a new seed to check that a fix is robust. The metric tiles compare each change with your previous setup.
Missions unlock in order. Each loads a broken configuration and awards up to three stars, usually for keeping batch throughput high or the bucket small once the interactive goal is met.
What the model assumes
Four app tenants send 1.5 requests per second each (2.5 times more between 40 and 70 seconds in a burst) with 300 to 1,500 prompt tokens and about 250 output tokens. The batch tenant sends 2,000 to 4,000 prompt tokens and about 500 output tokens. The agent sends 2 requests per second with context growing from 6,000 to 24,000 tokens and about 1,000 output tokens. Token limits reserve prompt plus 1,024 max_tokens and refund the unused part; request limits assume 1,500 tokens per request when converting the slider. Every tenant gets the same limit. The backend has four replicas of up to 48 sequences, a decode step of 6 ms plus 0.25 ms per sequence on each replica, and prefill counted as a tenth of a token per prompt token. The interactive goal is a queue wait under 2 seconds.
What it cannot tell you
Real tenants have different contracts, real clients retry, prompts cache, and real servers preempt and swap under memory pressure. Tokenizer differences, distributed rate-limit state and clock skew add error to real limits. Use the lab to see why token units, bucket size and queue order each matter, then measure your own gateway and backend with replayed traffic.
Read the full explanation
- Token-Based Rate Limiting for LLM APIs: Buckets, Fairness and Priority Lanes (companion article)
- Autoscaling LLM Servers: Why GPU Utilization Lies and Queue Depth Wins
- TTFT, TPOT and Goodput: Setting LLM Latency SLOs That Mean Something
All AI in Production labs · Lab catalog
- RFC 6585: Additional HTTP Status Codes (429 Too Many Requests)
- Fairness in Serving Large Language Models (Sheng et al., OSDI 2024)
- Envoy documentation: local rate limit filter
- Anthropic API documentation: rate limits
- OpenAI platform documentation: rate limits
Views are my own and do not represent any employer, past or present.