Why architecture matters here
GPUDirect Storage is architectural because the data path is a property of the whole node, not of any one library call. Whether a read can travel directly from an NVMe device into GPU memory depends on where those two devices sit on the PCIe tree: a peer-to-peer DMA transfer requires that the drive and the GPU can address each other across a PCIe switch without the transfer having to climb up to the CPU root complex and back down. On a well-designed GPU server the NVMe drives and the GPUs are deliberately placed under the same switch precisely so this path exists; on a general-purpose server they may not be, and then no amount of software enabling will produce a direct transfer. The bandwidth you can achieve is therefore decided at the hardware-layout level, long before any code runs.
The reason the bounce buffer is worth eliminating is that it taxes three scarce resources at once. It costs CPU cycles to drive the copy, it consumes host memory bandwidth — often the real constraint on a node where many GPUs and NICs are all contending for DRAM — and it adds latency by inserting an extra store-and-forward hop between disk and GPU. At small scale none of this matters; at the scale of a training job streaming a multi-terabyte dataset epoch after epoch, the host memory subsystem becomes a shared chokepoint that throttles every GPU on the node. Moving the data directly to HBM removes that contention entirely, which is why the benefit of GPUDirect Storage tends to grow, not shrink, as you add GPUs and push data rates up — exactly the regime that matters for large-model training and high-throughput inference over big embedding tables.
The third architectural fact is that correctness and speed here depend on the buffers, not just the path. A DMA engine cannot transfer into arbitrary application memory; it needs a buffer that is pinned (non-pageable, so the OS cannot move it mid-transfer) and registered with the driver so its physical addresses are known and stable, and reads generally must be aligned to device boundaries to hit the direct path. This pushes a discipline onto the application: allocate and register GPU buffers up front, reuse them, and issue aligned reads through cuFile rather than treating storage I/O as an afterthought. Get this wrong and the system does not error — it falls back to the slow path, silently.
That silent-fallback behavior is the single most important thing to understand about the architecture, because it inverts the usual relationship between correctness and performance. In most systems a misconfiguration announces itself as a failure you cannot miss; here a misconfiguration announces itself as nothing at all, because compatibility mode is a fully functional route that simply reintroduces the bounce buffer you were trying to avoid. Your program reads the right bytes and produces the right answers whether or not the direct path was taken, so a test that checks output tells you nothing about whether GPUDirect Storage is actually working. The only way to know is to measure — to instrument the fraction of bytes that traveled the direct path versus the compatibility path, and to compare achieved bandwidth against the fabric's ceiling. This is why running GPUDirect Storage well is fundamentally an observability problem layered on a hardware-topology problem: you must both build the node so the direct path exists and continuously verify that your workload is actually using it, because the failure mode of not using it is invisible to every check except a bandwidth measurement.
The architecture: every piece explained
Start at the top row, the direct read path. On one end sits the storage: an NVMe drive local to the server, or a remote drive exposed over NVMe-oF (NVMe over Fabrics) so that a whole rack of GPUs can pull from a shared high-speed storage tier. The application issues reads through the cuFile API, the CUDA library that, together with a kernel driver, understands how to program a direct transfer into GPU memory. The DMA engine then performs the actual peer-to-peer transfer, and the bytes land in the destination buffer in GPU HBM. The CPU set the transfer up but no host-memory copy of the data was made; the data flowed disk-to-GPU along the fabric.
The middle row shows what the direct path removes and requires. The CPU bounce buffer — the staging copy in host DRAM that the classic path depends on — is eliminated. The OS page cache is bypassed, the moral equivalent of opening the file with O_DIRECT so the kernel does not cache blocks in host memory on the way through. The PCIe switch is the piece of hardware that makes peer-to-peer possible: the drive and the GPU talk across it directly. When the storage is remote, a NIC with RDMA carries the NVMe-oF traffic, extending the same zero-CPU-copy discipline across the network so a read from a fabric-attached drive still lands directly in HBM.
The bottom row holds the two things that decide whether you actually get the direct path. Compatibility mode is the fallback: when the topology forbids peer-to-peer, the driver is missing, the buffer is unregistered, or the read is misaligned, cuFile transparently routes the transfer through a host bounce buffer instead of failing. It is functionally correct and performance-poor — the very thing you were trying to avoid — and because it is silent it is the chief operational hazard. Alignment and registration is the application's side of the bargain: GPU buffers must be pinned and registered with cuFile, and reads should be aligned to device boundaries, for the direct path to engage on every transfer rather than degrading to compatibility mode.
The ops strip underneath ties it together as an operating discipline. Verify the PCIe topology actually permits peer-to-peer between the specific GPUs and drives your job uses; register and reuse buffers rather than allocating per read; monitor the fallback (compatibility-mode) rate as a first-class metric so silent degradation surfaces; and size the storage fabric — local NVMe count, NVMe-oF network bandwidth — to the aggregate rate the GPUs can consume, because a direct path to an under-provisioned storage tier just moves the bottleneck from the CPU to the disks.
End-to-end flow
Follow a training job streaming its dataset. The data loader wants the next shard of training examples in GPU memory. It has, at startup, allocated a set of GPU buffers and registered them with cuFile, so the expensive pinning and registration happened once, not per read. To fetch a shard it calls the cuFile read API with a registered destination buffer, the file handle, and an aligned offset and length. cuFile and the kernel driver recognize that the drive and the GPU share a peer-to-peer path, program the DMA engine, and the shard flows from the NVMe device straight into HBM. The CPU issued the call and will be notified on completion, but it never copied a byte and the host memory bus stayed free for the other GPUs on the node.
Now scale it to a rack. The dataset is too large for local disks, so it lives on a shared storage tier reached over NVMe-oF. The read path is the same from the application's point of view — a cuFile read into a registered GPU buffer — but underneath, the request travels over the RDMA-capable NIC to the remote drive, which DMAs the data back across the fabric and into the GPU's HBM without a bounce through either the remote host's memory or the local host's memory. Eight GPUs on the node can each stream their own shards concurrently, and because none of those streams contends for host DRAM bandwidth, the node's aggregate storage throughput scales with the fabric rather than being capped by a single shared bounce buffer.
Consider what happens when a buffer is not registered. A developer adds a new preprocessing step that reads into a freshly allocated, unregistered GPU buffer. The cuFile call still succeeds and still returns the correct bytes — but under the hood the driver could not use the direct path, so it fell back to compatibility mode: it staged the data through a host bounce buffer and then copied it to the GPU, exactly the slow path GPUDirect Storage was meant to avoid. Nothing errors. The only visible symptom is that the job's storage throughput sagged and CPU utilization ticked up. If no one is watching the compatibility-mode metric, this regression can live in production indefinitely, silently taxing every epoch.
Now the topology failure that is decided before any code runs. The same job is scheduled onto a general-purpose server where the NVMe drives hang off a different PCIe root complex than the GPUs. Here a peer-to-peer transfer between drive and GPU is simply not possible — the electrical path forces the data up to the CPU and back down — so every read falls back to compatibility mode regardless of how carefully the buffers were registered. The application runs, the answers are correct, and the expensive GPUs spend a large fraction of each step waiting on a bounce-buffered storage path. The fix is not in software at all: it is to schedule the job onto nodes whose PCIe layout places drives and GPUs under a common switch, or to rebuild the storage tier so it does. This is why the operational playbook starts with verifying topology — the fastest cuFile call in the world cannot rescue a node where the direct path does not physically exist.