GPUDirect — Zero-Copy GPU I/O

One idea, four products — and the bounce buffer they delete

The unifying idea is narrow. A GPU exposes part of its device memory through a PCIe base address register — a window in the machine's physical address space that other devices on the bus can target with ordinary DMA. GPUDirect is the plumbing that lets a third party (another GPU, a network adapter, an NVMe controller) obtain addresses inside that window and read or write them directly. Everything else about the four members differs: different drivers, different APIs, different parts of the topology.

What they all delete is the same. Without GPUDirect, a block moves from an NVMe drive into a pinned buffer in host DRAM, then takes a second DMA across PCIe into the GPU; the network and GPU-to-GPU cases have the identical shape. That intermediate buffer costs an extra traversal of the root complex, twice the host memory bandwidth, CPU cycles to orchestrate and often to memcpy, and latency no queue depth fully hides. Delete the buffer and all four go with it — and if none of the four was your bottleneck, GPUDirect will not help.

The family at a glance

The figure below is the mental model to carry around. Three of the four members are data-path features: they change where the bytes physically land. The fourth, Async, is a control-path feature and does not appear in the same picture — it changes who initiates and completes the transfer, not where it goes.

Read it as a routing question, not a feature list: whatever sits on the far end of a transfer decides which member is in play.

P2P — GPU to GPU inside the node

Peer-to-peer is the oldest and most commonly relied-upon member: one GPU reads or writes another GPU's memory with no host involvement. In CUDA you ask cudaDeviceCanAccessPeer whether the pair supports it and call cudaDeviceEnablePeerAccess to turn it on; after that, a peer copy or even a direct pointer dereference from a kernel crosses the link.

Which link depends on the hardware. Where NVLink connects the pair, peer access rides the scale-up fabric and behaves close to a memory operation. Where it does not, the transfer goes over PCIe and becomes a property of the topology: both devices generally need to sit under a common switch or root complex, and PCIe Access Control Services on an intervening switch forces the traffic upstream and breaks the direct path outright. Lanes, switches and ACS belong to the PCIe treatment; the consequence belongs here — P2P is not a property of a GPU, it is a property of the pair.

RDMA — the network adapter writes straight into HBM

GPUDirect RDMA lets an RDMA-capable adapter (InfiniBand, or RoCE over Ethernet) place incoming payload directly into GPU memory and pull outgoing payload directly from it. The enabling piece is a kernel module — nvidia_peermem on current stacks, with newer kernels also supporting a dma-buf based route — that teaches the adapter's driver how to pin and translate GPU addresses so a memory region can be registered against device memory rather than host memory.

Topology bites hardest here, because the path involves two peripheral devices rather than one. An adapter and a GPU hanging off the same PCIe switch talk efficiently; a pair split across sockets forces traffic through the inter-socket link and can perform badly enough to erase the benefit. That is why large clusters are built rail-aligned, each GPU paired to a specific adapter. The fabric, the collectives and the rail design are covered in the InfiniBand article; the map entry is: peer is a NIC, module is nvidia_peermem.

Storage — NVMe straight into GPU memory

GPUDirect Storage completes the set by making a drive, or a remote filesystem client, the DMA peer. Reads land in GPU memory without a page-cache round trip. It is the member with the most moving parts: a kernel module (nvidia-fs), a user-space library reached through the cuFile API, a JSON configuration file, and a filesystem that is on the supported list — local NVMe or NVMe-oF, and specific distributed filesystems, not any mount you happen to have.

It is also the member with the most explicit fallback. The configuration carries a compatibility mode that, when a request cannot be served by the direct path, transparently performs a POSIX read into a host buffer and copies it to the GPU. Your call returns success and the right bytes either way. Alignment, threadpools, filesystem support and where the throughput comes from live in the dedicated GPUDirect Storage article; the map entry is: peer is a drive, API is cuFile.

Async — taking the CPU out of the control path too

The three paths above remove the CPU from the data path but leave it in the control path: the host still posts the work request, rings the adapter's doorbell, and polls for completion. In a latency-sensitive loop that residual CPU involvement, and the synchronization it forces between stream and host, can dominate what is left after the copies are gone.

GPUDirect Async addresses exactly that, letting communication operations be triggered and completed from the GPU's own execution stream — in the kernel-initiated variants, from inside a kernel — so a dependent send need not wait for the host to notice a kernel finished. It is the least universally available member and usually reaches you indirectly, through a communication library that opts into it. For the map, the distinction is the point: Async is not a fourth place for bytes to go, it is a different party issuing the instruction.

Preconditions, path by path

Each member fails for its own reasons. Keeping them in one table stops you from checking the wrong thing:

Two entries in the last column deserve emphasis because they are conditions, not bugs: an unloaded module, and a topology that never supported the path at all. Neither announces itself at runtime, and both are checkable before you run anything — which is why the routine below starts with static checks and only then looks at counters.

The failure mode is silence, not an error

This is the single most important operational fact about GPUDirect. Every one of these APIs is designed to keep working when the fast path is unavailable, because the alternative — an application that crashes on a machine with the wrong PCIe layout — would be worse. So the fallbacks are correct, transparent, and quiet.

Concretely: cudaMemcpyPeer succeeds whether or not peer access was ever enabled, staging through host memory when it was not. A cuFile read succeeds in compatibility mode with a POSIX read and a device copy behind it. A collective library that cannot register GPU memory with the adapter falls back to a host-staged transport and carries on. In all three cases your return codes are clean, your results are bit-identical, and your bandwidth is a multiple below plan. Absence of an error is not evidence the fast path engaged; it is evidence of nothing at all.

How to prove the fast path engaged

Verification comes in three layers, cheapest first. Static capability: is the machine even able to do this? nvidia-smi topo -m prints the connection matrix so you can see whether a GPU pair is NVLink-connected, under one switch, or crossing sockets, and whether a NIC is adjacent to the GPU you paired it with; lsmod tells you whether nvidia_peermem and nvidia-fs are loaded; the storage package ships a checker that reports driver status and which mounts are supported.

Runtime evidence: ask the software what it chose. Collective libraries log their selected transport when debug output is on, naming explicitly whether a GPU-direct network path or a host-staged one was used. The storage stack exposes counters separating bytes served by the direct path from bytes served in compatibility mode — a nonzero compat counter is the answer on its own. For P2P, query peer access rather than assuming the enabling call was made.

Physical evidence, the layer that cannot be argued with: on a real fast path, host DRAM traffic and CPU utilization do not scale with transfer volume, and a system-level profile shows no host-to-device copies shadowing the transfer.