RDMA for GPUs — Remote Direct Memory Access

Why it matters

Multi-node training scales rely on RDMA. Understanding enables performant distributed training.

The architecture

Traditional network: user → kernel → NIC → wire → NIC → kernel → user. Many copies.

RDMA: user → NIC → wire → NIC → user memory. Zero copies.

How it works end to end

Infrastructure: InfiniBand or RoCE (RDMA over Converged Ethernet). Mellanox/NVIDIA NICs.

NCCL: uses RDMA + GPUDirect for GPU collective communication.

Latency: sub-microsecond within same fabric.