Why it matters

NCCL powers multi-GPU training. Foundation of distributed frameworks.

Advertisement

The architecture

All-reduce, all-gather, broadcast.

Ring, tree algorithms.

NVLink + IB aware.

NCCL libraryCollectivesall-reduce, gatherAlgorithmsring, treeFabric awareNVLink + IB + P2PPyTorch DDP + Megatron + DeepSpeed all use NCCL under hood
NCCL library.
Advertisement

How it works end to end

PyTorch DDP.

DeepSpeed.

Megatron.