A mixture-of-experts model breaks the assumption every dense-serving intuition rests on: that one parameter count sets both the memory bill and the compute bill. In an MoE a token touches a small fraction of the feed-forward weights, so the FLOPs look like a far smaller model — but the router is free to send the next token anywhere, so every expert must be resident in HBM somewhere in the cluster before the request arrives. You pay dense-model memory for sparse-model compute, then have to keep the GPUs you bought for capacity busy. This is the GPU-side view: where experts live, what dispatch and combine cost, why routing skew becomes a straggler, and how batching interacts with expert locality.
The inversion: cheap FLOPs, expensive residency
Take a model with 256 experts and top-8 routing. Each token runs through 8 of the 256 expert MLPs, so roughly 3% of the feed-forward parameters do work on it. None of that reduces what you must hold: routing is decided per token, per layer, at runtime, so all 256 experts have to be loaded and reachable at every layer, all the time.
That reshapes capacity planning. A dense deployment is sized by throughput. An MoE is sized by weight residency first: how many GPUs it takes to hold the checkpoint at your serving precision, with room left for activations and KV cache. Only then do you ask whether those GPUs can be kept busy. A compute-bound MoE deployment is a rarity; most are bound by memory, by all-to-all, or by the slowest GPU in the expert group.
Memory footprint versus active FLOPs
Expert weights dominate the checkpoint, commonly the large majority of parameters in a modern sparse model, while attention, embeddings and the router are comparatively tiny. Expert weights are pure residency cost: they occupy HBM whether or not the current batch routes to them. The attention blocks, meanwhile, are entirely dense and behave exactly as they would in a non-MoE model, KV cache included.
The two compete for the same HBM. Every gigabyte of expert weights on a GPU is a gigabyte that cannot hold KV cache, and KV capacity is what sets maximum concurrency. Adding GPUs relieves both, but widens the all-to-all, so the relief is not free. Quantising experts harder than attention is the standard lever here. The active-parameter figure on the model card tells you almost nothing about how many GPUs you need.
Expert placement: how experts land on GPUs
Expert parallelism is the placement axis MoE adds: shard the expert set by index across a group of GPUs, so 256 experts over 32 GPUs gives 8 experts per GPU, each owning its slice outright. It is orthogonal to tensor parallelism, which splits a single weight matrix, and composes with it: large deployments run tensor parallelism in the attention blocks and expert parallelism across the MoE blocks.
Placement is a topology decision. Dispatch traffic flows between every pair of GPUs in the expert-parallel group, so the group’s span decides which links carry it. A degree that fits inside one NVLink domain keeps the exchange on the fast intra-node fabric; the moment the group spans nodes, part of every dispatch and combine crosses InfiniBand or Ethernet at a fraction of the bandwidth and several times the latency. Picking the expert-parallel degree is mostly a question of where the node boundary falls.
Dispatch and combine: two all-to-alls per layer
Once experts are spread across GPUs, tokens must travel to them. Dispatch is an all-to-all: each GPU packs the tokens whose selected experts live elsewhere, and receives in return every token in the global batch that routed to one of its own experts. Each GPU runs its local experts, and combine is the mirror-image all-to-all returning each output to the rank that held the token, where the router weights sum the top-k results.
That is two collectives per MoE layer: in models where nearly every block is an MoE block, roughly twice the layer count of collectives on the critical path of every forward pass. The algorithms, and the NVLink-versus-InfiniBand behaviour, belong to NCCL collectives; what differs here is the frequency, and the fact that the expert GEMMs cannot start until dispatch has finished.
Why the decode all-to-all is latency-bound
The instinct is to treat all-to-all as a bandwidth problem. During prefill it is: thousands of tokens per request, multiplied by top-k, make genuinely large buffers. During decode it is not. Each sequence contributes one token per step, so the payload is batch × k × hidden × bytes-per-element, then split across the peers. Illustratively, a 128-sequence batch with top-8 routing, a hidden size of a few thousand and 8-bit activations produces single-digit megabytes in total; split among dozens of ranks, each message is tens of kilobytes.
Messages that small never reach the bandwidth-limited regime. Their cost is per-hop latency, kernel launch overhead, and everyone waiting for everyone. Widening expert parallelism therefore buys sync points rather than bytes, and serving stacks increasingly replace the generic collective with fused, one-sided put/get kernels. CUDA-graph capture of the decode step matters for the same reason: launch overhead is a first-order term at these sizes.
Routing skew, stragglers, and the max-not-mean problem
Training-time load balancing aims for a router that spreads tokens evenly. Serving traffic does not cooperate: code-completion and conversational requests induce visibly different expert distributions, and one tenant sending homogeneous prompts can concentrate load on a few experts for minutes.
The damage is structural rather than proportional. Combine cannot begin until every rank has finished its local experts, so the layer’s wall-clock cost is set by the most loaded GPU, not the average one. A group where one GPU receives twice the mean token count runs at roughly half its potential utilisation no matter how idle the others are. The skew is redrawn every layer and every step, so it never averages out: you take the maximum, repeatedly. Instrument a per-expert token histogram and the ratio of maximum to mean per-GPU load; that ratio multiplies MoE-layer latency directly.
Capacity factors and dropped tokens at inference
Fixed-shape kernels want to know in advance how many tokens each expert will receive, so the classic implementation gives each expert a buffer of capacity factor × (tokens × k / experts). Tokens beyond that capacity are dropped: they skip the expert and pass through on the residual alone. Padding, in the other direction, runs GEMMs over empty rows.
Dropping is tolerable in training, where it acts as noise the model learns around. At serving time it is a defect: whether your token is dropped depends on who else is in the batch, so the same prompt with the same seed can produce different output depending on its batchmates. That breaks reproducibility and makes quality regressions miserable to debug. Production stacks therefore favour dropless formulations, expressing the ragged per-expert token counts as a block-sparse or grouped GEMM over variable-sized tiles.
Batching and expert locality
Batching an MoE is subtler than batching a dense model, because routing immediately divides the batch. B tokens with top-k over E experts give each expert about B×k/E tokens on average. With 256 experts, top-8 routing and a batch of 64, the average expert sees two tokens: a grouped GEMM with an M dimension of two, against tensor-core tiles that want dozens to hundreds of rows.
So the expert GEMMs run deep in the memory-bound regime: the kernel streams a full expert’s weights out of HBM to serve a handful of tokens, and arithmetic intensity collapses. An MoE needs a much larger batch than a dense model of the same active size to reach comparable efficiency, roughly E/k times larger, to restore the same per-matrix token count. That argues for aggressive continuous batching, but the pressure is two-sided: a larger batch also puts more bytes through dispatch and combine, and more KV cache in competition with the expert weights.
Replicated experts and dynamic rebalancing
If a few experts are persistently hot, replicate them: place a second copy on another GPU and let dispatch spread that expert’s tokens between the copies. Inference makes this cheap in a way training does not: weights are read-only while serving, so replicas never need reconciling, and the cost is the duplicated HBM plus a richer routing table.
The extension is dynamic rebalancing: profile the per-expert histogram over a window, compute a placement that levels predicted per-GPU load, and migrate experts to match. The constraint is that moving expert weights is expensive and cannot happen mid-step, so rebalancing runs on a cadence of minutes and works only against traffic whose routing distribution is stable over that horizon. When the mix shifts faster than you can migrate, a static placement from offline profiling beats chasing the distribution.
Prefill and decode want different expert parallelism
The two phases stress the machinery in opposite directions. Prefill processes thousands of tokens at once: every expert is hit, grouped GEMMs are well filled, all-to-all payloads are large enough to be bandwidth-bound, and the layer is genuinely compute-heavy. Decode processes one token per sequence: expert GEMMs are skinny, messages are tiny and latency-bound, and imbalance shows starkly because there are too few tokens to average over.
No single configuration is optimal for both, which makes prefill/decode disaggregation a natural fit: separate GPU pools with independently chosen expert-parallel degrees and batch policies. Either way the instrumentation list is the same: per-expert token counts, all-to-all time as a fraction of step time, per-GPU idle time inside the MoE layer, and drop rate if you have not gone dropless. Those four numbers explain nearly every MoE serving problem you will hit.