Core concept
NVIDIA Quantum IB switches support. All-reduce data flows through switch which aggregates. Faster + lower CPU load.
Advertisement
How it works
Requires SHARP-capable topology. Config complex.
Advertisement
Trade-offs + gotchas
Large-scale distributed training. Reduces network hop for all-reduce.