Evidence label: deterministic synthetic model. This page runs a small JavaScript model of one MoE layer. It generates a pool of synthetic token vectors from eight topics and a router with one weight vector per expert, scores every token against every expert, picks the top k, and enforces a per-expert capacity or runs dropless. A bias-based balancer adjusts expert biases after each of 24 warm-up batches, then eight measured batches report drops, imbalance, rerouting and an illustrative layer time under eight-way expert parallelism. Nothing here runs a real model or measures a GPU; all numbers are calculated estimates.
Play the lab
How to play
Choose the number of experts, top-k, overflow handling, capacity factor, balancing step γ and tokens per batch. Every change re-runs the same seeded traffic, so differences come from your change. Watch the tiles: dropped assignments, the busiest expert's load relative to the mean, layer time, a quality proxy, how many tokens balancing rerouted away from their first choice, and how many capacity slots stayed empty. Events skew the router toward two experts or concentrate traffic on one topic.
Missions unlock in order; each loads a broken setup. Stars reward the more efficient fix, such as lower layer time or higher quality.
What the model assumes
Tokens are eight-dimensional vectors scattered around eight topic centres; each expert's router vector leans toward one topic, with noise, and is normalised. Router scores are dot products. Capacity is ceil(factor × tokens × k / experts), filled in arrival order. The balancer subtracts γ times each expert's relative overload from its bias after every batch, as in auxiliary-loss-free balancing. Layer time is 2 ms plus 0.0035 ms per token-slot on the busiest GPU (padded slots in capacity mode, actual tokens when dropless) plus an all-to-all term; these constants are illustrative. The quality proxy subtracts penalties for partially and fully dropped tokens, for rerouted tokens and for top-1 routing.
What it cannot tell you
A real router co-adapts with its experts during training, so heavy balancing in a real model costs less specialisation than in this fixed synthetic router. Real step times depend on kernels, interconnect bandwidth and overlap of communication with compute. The quality proxy is not an evaluation of any model. Use the lab to understand the shape of the trade-offs, then measure routing statistics and latency on your own stack.
Read the full explanation
- Mixture-of-Experts Routing: Load Balancing, Capacity Factors and Dropped Tokens (companion article)
- Tensor or Pipeline Parallelism for Serving: Choosing How to Split a Model
- Grouped-Query Attention: How Sharing KV Heads Shrinks Inference Memory
All AI in Production labs · Lab catalog
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017)
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (Fedus et al.)
- ST-MoE: Designing Stable and Transferable Sparse Expert Models (Zoph et al., 2022)
- MegaBlocks: Efficient Sparse Training with Mixture-of-Experts (Gale et al., 2022)
- DeepSeek-V3 Technical Report (auxiliary-loss-free load balancing)
- Mixtral of Experts (Jiang et al., 2024)
Views are my own and do not represent any employer, past or present.