A modern mixture-of-experts layer looks the way it does because of a chain of decisions made under systems pressure, not because anyone derived it from first principles. GShard established the shape: a router, expert feed-forward blocks scattered across devices, and a fixed-size buffer per expert so the layer could be expressed as static-shape collectives. Switch Transformer then made the lineage’s most consequential simplification—route each token to exactly one expert. Almost everything that followed, from shared always-on experts to dropless and expert-choice routing, is a targeted repair of a cost those two papers accepted. Read as design history, with the systems consequence of each choice in view: how much crosses the fabric, what shape the buffers take, and whether the expert kernel stays regular.
What GShard established: the static-shape contract
The hard problem in a sparse layer is not the math, it is that routing is data-dependent. How many tokens want expert 7 is unknown until the router runs, and it changes every step. GPUs and their collectives hate that: kernels want shapes known at launch, and an all-to-all wants every rank to agree in advance on how many bytes it will send and receive.
GShard’s contribution was the contract that makes this tractable. Give every expert a fixed-size buffer, sized once from the batch, and the dynamic problem collapses into a static one: every rank sends the same number of slots to every other rank, the expert’s feed-forward block runs on a tensor of known shape, and no synchronising count exchange precedes the transfer. The routing itself was top-2—each token picks its two highest-scoring experts, is processed by both, and the outputs are mixed by the normalised gate weights. Top-2 plus a fixed buffer is the template every later design inherits or deliberately breaks.
Switch's argument: one expert is enough
Switch Transformer’s central claim was that the second expert does not earn its cost. Set k = 1: each token goes to exactly one expert, whose output is scaled by the router’s probability for it. The prevailing intuition had been that top-2 was necessary for the router to learn at all—you need to compare experts to know which is better. Switch showed a top-1 router trains fine, given the stability repairs described below.
The systems payoff is why the choice stuck. Halving the number of (token, expert) assignments halves the dispatch and combine payload and halves expert-side FLOPs for a given batch. It also simplifies the kernel: with one destination per token the permutation is a plain gather, no per-token weighted sum over two expert outputs needs accumulating, and the combine step is a single scale. For a fixed budget that saving buys more experts or a bigger batch—both worth more than the second expert was.
Capacity factor: fitting a dynamic problem into a static buffer
A fixed buffer needs a size, and that is the capacity factor — the multiplier on the perfectly-balanced share. Capacity 1.0 gives each expert exactly enough slots for its fair fraction of the batch; anything above is deliberate headroom for skew. It is best read as a systems knob, not a hyperparameter: the number that converts ‘however many tokens happen to want this expert’ into a buffer shape you can compile against.
Everything downstream reads that shape. The all-to-all message size is capacity × hidden size × experts-per-rank. The expert GEMM has a fixed M dimension, so it can be a batched or grouped matmul with no host round-trip to discover sizes, and activation memory is allocated once and reused. Raise capacity and you buy tolerance for imbalance with linearly more bytes on the wire and linearly more padded FLOPs in a GEMM whose extra rows are zeros; lower it and the buffers overflow. The arithmetic belongs to the transformer-math side; the point here is that one scalar sets both the layer’s memory shape and its communication volume.
Token dropping: the consequence nobody wanted
If a buffer is fixed and more tokens route to an expert than it has slots, the surplus tokens are dropped — they skip the expert entirely and pass through the residual connection unchanged. This is not an error path; it is the designed behaviour, and it is the direct price of the static-shape contract.
Two things make it worse than it sounds. Routing skew is not uniform noise: routers develop genuine preferences, so the same few experts overflow repeatedly and the same kinds of token get dropped repeatedly. And the imbalance that hurts is the maximum over experts, not the mean — because buffer size is uniform, one hot expert forces the capacity factor up for everyone. That is what motivated the load-balancing auxiliary loss: a term pushing the router toward even assignment, bought by routing slightly against the model’s preference. This uncomfortable trio — padding waste, dropped tokens, a loss fighting the router — is what later ‘dropless’ work set out to eliminate.
The stability fixes a sparse router needed
Switch did not get top-1 routing to train by changing the routing alone. It needed three small numerical repairs, and they are worth knowing because the lineage kept all three.
First, compute the router in fp32 even when the rest of the model runs in reduced precision. The router is a tiny matmul from hidden size to expert count plus a softmax, so a local fp32 cast costs essentially nothing — but a softmax over expert logits is exactly where low-precision rounding flips an argmax, and a flipped argmax sends a token to a different device. Second, scale down the initialisation: smaller initial weights keep early logits narrow, so the router does not commit hard to one expert before it has learned anything. Third, expert dropout — a higher dropout rate inside the expert blocks than in the shared layers, because each expert sees only a fraction of the tokens and overfits its slice faster than a dense layer would.
Shared experts: putting the common case back
The first big post-Switch change was to stop routing everything. In a pure top-1 layer, knowledge every token needs must be replicated into every expert, because any given token sees only one of them. That is a quiet waste of parameters, and it makes the router’s job harder: it is asked to specialise on a signal partly made of generic material.
The repair is a small number of shared, always-on experts that process every token unconditionally, alongside the routed ones, leaving the routed experts free to hold what is genuinely specialised. Systems-wise this is cheap and even helpful: a shared expert is a dense feed-forward block, so it is replicated rather than sharded, needs no dispatch or combine, and can run while the routed path’s communication is in flight. It gives the scheduler real compute to overlap with the collective — otherwise one of the hardest gaps in an MoE layer to fill.
Fine-grained experts: more of them, each smaller
Switch-era designs used relatively few, relatively fat experts. The later trend is the opposite: split the same parameter budget into many more, much narrower experts and raise the number selected per token to compensate. The argument is combinatorial — with more experts to choose from and several chosen, the number of distinct expert combinations a token can route through grows enormously, so the layer expresses finer specialisation at the same active parameter count.
The systems bill lands on the kernel, not the wire. Each expert’s matmul is narrower, so the same total FLOPs spread over many smaller GEMMs that individually struggle to fill a GPU. This is precisely why grouped and batched GEMM kernels became standard equipment for MoE: one kernel launch handles many independent small matmuls, keeping the SMs occupied where a loop of separate launches would leave them idle. Fine granularity is a modelling win that only became affordable once the kernel side caught up.
Dropless routing: paying in shape instead of tokens
Dropping tokens to protect a buffer is an odd trade, and the dropless approach refuses it: keep the assignment exactly as the router chose it and make the computation handle a variable number of rows per expert, instead of truncating the assignment to fit a fixed one.
Concretely, the expert forward pass becomes a block-sparse or grouped operation over a ragged batch — rows grouped by destination expert, the kernel walking a set of variable-length groups instead of a rectangular tensor. Nothing is padded to a common capacity and nothing is discarded. The costs move rather than vanish. Message sizes become dynamic, so the collective needs a count exchange before the payload, an extra synchronisation the static-shape design existed to avoid. And the kernels are more specialised: no falling back on a plain dense batched matmul. What you get back is a layer whose numerics no longer depend on a capacity hyperparameter at all, removing a class of tuning and a class of silent quality loss.
Expert-choice routing: inverting the assignment
The most conceptually clean fix inverts the direction of the choice. Instead of each token selecting its top experts, each expert selects its top-scoring tokens from the batch, up to its capacity. The scores come from the same router matrix — only the axis you take the top-k along changes.
Load balance then stops being something you hope for and becomes structural. Every expert takes exactly its capacity, so no buffer overflows, no token is dropped, the auxiliary balancing loss is unnecessary, and the expert GEMM has no padded rows. The cost is a different asymmetry: a token can now be picked by several experts or by none, so per-token compute is uneven — fine for training, awkward for autoregressive decoding, where the selection would run over a batch of single tokens and let one sequence’s routing influence another’s. Hence expert-choice appears far more on the training side than in serving.
Reading a modern MoE as accumulated decisions
Put the lineage end to end and a contemporary sparse layer stops looking arbitrary. The fixed per-expert buffer and the all-to-all pair around the expert block are GShard’s static-shape contract, kept because it is what compiles into regular kernels. Low-k routing with a gate scale is Switch’s bet that the marginal expert is not worth its bytes; the fp32 router cast is a numerical scar from that bet. A handful of always-on experts is the admission that not everything should be routed. Many narrow experts driven by a grouped GEMM is the modelling win that arrived once the kernels could carry it. Dropless and expert-choice are two escapes from the capacity trade — one paying in kernel complexity and dynamic messages, the other in uneven per-token compute.
Read a new MoE paper against that list and the novel part stands out immediately: it is whichever decision the authors chose to reverse, and the interesting question is always which systems cost they took on in exchange.