Why architecture matters here
MoE fails at serving in specific ways: hot experts overload while others idle; all-to-all bandwidth saturates on modest clusters; capacity overflow drops tokens and degrades output. Each failure has an architectural fix.
The architecture matters because MoE's payoff — many parameters, small compute per token — only holds if the routing and all-to-all machinery work well at your scale. Small clusters may not fit; expert parallelism dictates GPU topology needs.
With the pieces in mind, you can size cluster + capacity + drop policy against latency and quality SLOs.
The architecture: every piece explained
The top strip is the routing path. Input tokens arrive as a batch × seq tensor. Router is a small MLP that emits a top-K expert selection per token. Expert group is the set of MLPs sharded across GPUs. All-to-all dispatches tokens to their chosen experts across the fabric.
The middle row is the compute + safety. Expert compute is a sparse GEMM inside the receiving expert. Combine sums weighted outputs and returns tokens to origin. Capacity + drop: each expert has a per-batch capacity; overflow tokens are dropped or spilled based on policy. Load balance loss is added during training to keep utilization even; at serving time it shows up as observed imbalance.
The lower rows are ops. Expert parallelism spreads experts across GPUs (usually distinct from tensor + data parallel groups). Metrics track expert utilization and drop rate. Ops plans capacity, places experts to minimize all-to-all distance, and handles fault cases per-expert.
End-to-end flow
End-to-end: an inference request arrives. Router computes top-2 expert selection for each token. All-to-all sends tokens to their expert GPUs. Experts compute; combine returns results. Some tokens hit expert 3 which is at capacity; policy is to drop-and-use-residual. Metrics show 0.6% drop rate — within budget. Expert utilization ranges from 0.7 to 1.0 across experts — imbalanced but not catastrophic. Total latency is 42 ms; all-to-all is 18 ms of that. Compared to dense of same effective quality, serving cost is 40% lower.