Why it matters

MoE models scale differently. Understanding expert parallelism shapes modern architectures.

Advertisement

The architecture

Experts partitioned across GPUs.

Routing: token → expert. All-to-all shuffle to send tokens to correct GPU.

Expert parallelismExperts distributeddifferent GPUsAll-to-allroute tokensPer-expert computethen all-to-all backLoad imbalance: some experts get more tokens; auxiliary loss balances
MoE parallelism.
Advertisement

How it works end to end

Communication: all-to-all shuffle both directions. Expensive.

Load balancing: aux loss encourages even routing.

Combine with other parallelism: expert + tensor + data + pipeline.

Frameworks: Megatron-LM, DeepSpeed MoE, Tutel.