Why it matters

Comm math shapes scaling efficiency. Understanding shapes design.

Advertisement

The architecture

All-reduce volume: 2 * (N-1)/N * bytes.

Time = volume / effective BW.

Communication mathData volume2*(N-1)/NBandwidthGB/s per linkTimevolume / BWRatio time_comm / time_compute determines scaling efficiency; overlap essential
Comm math.
Advertisement

How it works end to end

All-reduce time: 2(N-1)/N * data / BW.

Ring: bandwidth-optimal.

Latency floor: message-count * latency.

Overlap with compute.