Evidence label: deterministic synthetic model. This page runs a small JavaScript model of one LLM replica split across GPUs. It computes weight and KV cache memory per GPU, a decode step as weight and KV reads at a fixed memory bandwidth plus ring all-reduces whose cost depends on the link, pipeline stages run in sequence with micro-batches, and a prefill cost from a fixed compute rate. It then finds the concurrency each replica needs for the offered load. No model or GPU is involved; every number is a calculated estimate.
Play the lab
How to play
Pick a model and precision, the number of GPUs, and the tensor and pipeline sizes. The number of replicas is the GPU count divided by TP × PP; leftover GPUs sit idle but still cost money. Choose the link inside a node and the request rate. Tensor sizes above 8 cross the node boundary and use a slower network. The timeline shows one replica; the tiles show the whole deployment. Inject the traffic or long-prompt events to stress a configuration.
Missions unlock in order. Each loads a broken configuration and awards up to three stars, for fewer GPUs, more capacity or lower latency once the goal is met.
What the model assumes
Each GPU has 80 GB with 90% usable and 4 GB of runtime overhead, 2.5 TB/s of effective memory bandwidth and 600 TFLOP/s of effective compute. NVLink-class links are modelled at 300 GB/s with 3 µs per algorithm step, PCIe at 20 GB/s and 25 µs, and the inter-node network at 40 GB/s and 10 µs. All-reduce uses a ring cost with a logarithmic latency term; two run per layer. Prompts are 2,000 tokens (8,000 with the event) and outputs 300. KV cache is 16-bit with 8 KV heads of dimension 128. Prefill compute is subtracted from decode capacity; load per replica varies by 25% to estimate peak concurrency.
What it cannot tell you
Real engines overlap communication with compute, use custom all-reduce kernels, fuse operations, and schedule prefill and decode together, so absolute numbers will differ. Uneven stages, embeddings, attention cost at long context, mixture-of-experts routing and topology details are absent. Use the lab to understand which way each choice pushes latency and throughput, then benchmark candidate splits on your own hardware.
Read the full explanation
- Tensor or Pipeline Parallelism for Serving: Choosing How to Split a Model (companion article)
- KV Cache Capacity Planning: How Many Concurrent Users Fit on One GPU
- Continuous Batching Explained: Why Static Batches Leave GPUs Idle
All AI in Production labs · Lab catalog
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (Shoeybi et al., 2019)
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (Huang et al., 2018)
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (Narayanan et al., SC 2021)
- Efficiently Scaling Transformer Inference (Pope et al., MLSys 2023)
- vLLM documentation: distributed inference and serving
Views are my own and do not represent any employer, past or present.