Why it matters
Communication is often the bottleneck in distributed training. Understanding the topology and NCCL primitives helps you choose parallelism strategies (data, tensor, pipeline) that fit the hardware.
The architecture
NVLink: direct GPU-to-GPU high-bandwidth link. Fifth-generation NVLink on H100/B100 gives 900 GB/s. Point-to-point between adjacent GPUs.
NVSwitch: fabric that connects up to 8 GPUs in a fully-connected topology. All GPUs can talk to all others at NVLink bandwidth.
How it works end to end
NCCL primitives: all-reduce (sum values across GPUs, distribute result), all-gather (each GPU has all values), broadcast, reduce-scatter. Implemented on top of NVLink/InfiniBand.
Ring all-reduce: divides work across N-1 steps, achieves nearly optimal bandwidth. Standard for data-parallel training.
Tree all-reduce: log-depth reductions, better for small messages or complex topologies.