Expert Parallel Math Deep

Why it matters

Expert parallel shapes MoE. Understanding shapes design.

The architecture

Experts sharded across GPUs.

All-to-all: send tokens to expert GPUs.

Compute + return.

How it works end to end

All-to-all is bottleneck.

Load balance affects.

Combine with TP + PP.

Communication overlap.