Why architecture matters here
MoE training fails on load balance and router stability. Without auxiliary losses, the router collapses onto a few experts. Without careful capacity + jitter, tokens get dropped in ways that destabilize training. The architecture matters because these are joint training + model design issues, not just serving concerns.
With the pieces mapped, you can pick top-K, capacity factor, and auxiliary losses that produce a healthy MoE.
The architecture: every piece explained
The top strip is the forward path. Token input arrives. Gate MLP emits a softmax over experts per token. Top-K selection picks the top-K experts. Dispatch sends the token to the selected experts via all-to-all in distributed setups.
The middle row is the compute + regularization. Expert MLPs are the specialized feed-forward blocks. Combine sums outputs weighted by gate scores. Load balance loss is an auxiliary loss that penalizes uneven expert utilization. Router z-loss stabilizes the softmax by penalizing large logits.
The lower rows are the systems. Sparse GEMM kernels execute expert compute efficiently. Expert parallelism shards experts across GPUs. Training uses capacity factor (typically 1.25-1.5x) and jitter (random noise added to gate scores) to prevent overload and encourage exploration.
End-to-end flow
End-to-end: during training, a batch flows through the model. Gate emits per-token expert scores; top-2 selected. Dispatch routes tokens. Experts compute. Combine reduces. Total loss = task loss + load balance loss + router z-loss. Backprop updates all experts + the gate. Over training, experts specialize (some for code, some for math). Load balance auxiliary loss prevents collapse; router z-loss keeps softmax stable. Capacity factor of 1.25 means some tokens dropped under high routing skew — an acceptable trade.