Why it matters

EP is MoE-specific parallelism. Understanding shapes MoE design.

Advertisement

The architecture

Experts sharded across GPUs.

All-to-all: send tokens to their experts.

Compute experts locally.

All-to-all: send back.

Expert parallelism flowExpertsshardedAll-to-allroute tokensCompute + returnexpert outputsAll-to-all is the bottleneck; capacity factor tunes load balance
EP.
Advertisement

How it works end to end

All-to-all: heavy communication.

Capacity factor: balance load.

Combine with DP + TP + PP.