Why architecture matters here
RDMA is architectural because it relocates the boundary between hardware and software in the data path. In the conventional model the kernel is a mandatory intermediary: it owns the network stack, copies data, enforces protection, and schedules the CPU work of sending and receiving. RDMA moves that responsibility into the NIC and lets the application talk to the hardware almost directly, which is what eliminates the per-message system call, the data copies, and the CPU processing that dominate the cost of high-rate networking. But relocating that boundary means the protections and services the kernel used to provide — memory safety, flow control, reliability — must now be provided some other way, by the NIC and the fabric, and understanding RDMA is largely understanding how those services are re-provided outside the kernel.
Memory registration is the first consequence, and it is not optional overhead — it is the mechanism that makes safe kernel bypass possible. For a NIC to DMA data directly into or out of application memory, two things must be true: the memory must be pinned so the operating system cannot move or page it out while the NIC is using it, and the NIC must have permission and address translation for that region. Registration accomplishes both, producing keys that authorize access to the region. This is why registration is expensive and why it must happen before the fast path: it is the setup that lets the NIC touch application memory safely without the kernel mediating each access. The architectural implication is that you register buffers up front and reuse them, because paying registration cost on every transfer would give back the very overhead RDMA exists to remove.
The one-sided operations are what make RDMA qualitatively different from message passing, and they are the source of its lowest latencies. In a two-sided model both ends participate: the sender posts a send, the receiver must have posted a matching receive, and both CPUs are involved at the endpoints even if the transfer itself is offloaded. In a one-sided RDMA read or write, only the initiator is involved: it can write data straight into a remote machine's registered memory, or read data straight out of it, without the remote CPU being notified or involved at all. The remote application's memory is modified by the remote NIC on the initiator's behalf. This is extraordinarily powerful — it enables things like remote data structures and disaggregated memory — and it is also a sharp edge, because it means correctness and security now depend on registration keys and careful coordination rather than on the remote CPU checking each message.
The deepest operational truth about RDMA — the one that surprises teams who treat it as a drop-in speedup — is that the dominant transport for deploying it over Ethernet, RoCEv2, requires the network fabric to be effectively lossless, and this requirement reshapes how you must run the network. RDMA's reliable transport was designed assuming packet loss is rare, so it handles drops far less gracefully than TCP does; a lossy fabric that TCP would ride through comfortably can collapse RDMA throughput. To make Ethernet lossless enough, operators deploy Priority Flow Control (PFC), which lets a congested switch pause upstream senders instead of dropping their packets, and Explicit Congestion Notification (ECN), which marks packets to signal congestion so senders slow down before queues overflow. The catch is that PFC's pausing is a blunt instrument: pause frames propagate backward through the fabric, so congestion at one switch can spread to others, and in pathological topologies PFC can even deadlock. This means running RDMA at scale is as much about tuning congestion control and flow control on the fabric as it is about the RDMA endpoints — the fabric is part of the architecture, not a neutral pipe, and the most common serious RDMA incidents are congestion-spread and PFC storms, not endpoint bugs.
The architecture: every piece explained
The top row is the endpoint stack. The application initiates transfers by posting work requests — descriptions of what to send, receive, read, or write. It does this through the verbs API, the RDMA programming interface, which is built around queue pairs: each connection has a send queue and a receive queue onto which work requests are posted. The RNIC — the RDMA-capable NIC, also called a host channel adapter — is the hardware that actually performs the transfers, reading from and writing to registered memory and moving data across the fabric. The essential shape is that software posts requests to queues and the NIC does the work, rather than software doing the work through the kernel.
The middle row is what makes the fast path fast and safe. Memory registration pins application buffers and produces keys so the NIC can DMA into them directly without the OS moving the pages out from under it. Kernel bypass is the result: once buffers are registered and queue pairs set up, posting a transfer does not cross into the kernel per operation, eliminating the system-call and copy overhead. The RDMA read/write operations are the one-sided transfers — the NIC reads or writes remote registered memory directly, without the remote CPU's involvement. And the completion queue is how the initiator learns a transfer finished: rather than an interrupt per operation, it polls the completion queue for completion entries, which keeps latency low on the fast path.
The third row is the transport and the fabric it demands. RoCEv2 — RDMA over Converged Ethernet, version 2 — is the transport that carries RDMA over ordinary UDP/IP networks, which is what lets RDMA run on standard datacenter Ethernet rather than requiring a specialized fabric. But RoCEv2's reliability assumes almost no packet loss, so it needs a lossless fabric, built from Priority Flow Control (which pauses senders at a congested switch instead of dropping packets) and Explicit Congestion Notification (which marks packets so senders throttle before queues overflow). These two mechanisms are what turn a normally-lossy Ethernet into the near-lossless environment RDMA needs.
The ops strip names the disciplines that keep RDMA healthy. Register buffers up front and reuse them so the registration cost is paid once, not per transfer. Watch PFC and ECN activity closely, because pause frames and congestion marks are the early signals that the lossless fabric is under stress and at risk of spreading congestion. Size queue pairs and completion queues for the connection count and in-flight work so they do not exhaust. And guard against congestion spread — the characteristic way RDMA fabrics fail — by tuning congestion control and being alert to PFC propagating backpressure across the network.
End-to-end flow
Follow a single transfer on a healthy fabric. Two machines have established a connection with a queue pair on each side, and both have registered the memory buffers they will use. The initiator wants to write a block of data into a specific registered region of the remote machine's memory — it holds the remote address and the access key from an earlier exchange. It posts an RDMA write work request to its send queue describing the local source buffer, the remote destination address, and the key. Its RNIC reads the data from local registered memory and sends it across the fabric; the remote RNIC receives it and DMAs it straight into the specified remote memory region — the remote CPU is never interrupted and never touches the data. The initiator polls its completion queue, sees the write completed, and moves on. Microseconds elapsed, no system calls, no copies, no remote CPU involvement. This is the design delivering exactly what it promises.
Now the registration cost, and why it must stay off the fast path. Suppose a naive implementation registers a fresh buffer for every message and deregisters it afterward. Registration is a heavyweight operation — it pins pages and sets up NIC address translation — so doing it per transfer injects exactly the kind of overhead RDMA was meant to eliminate, and the fabric sits idle while the CPU churns through registrations. The correct pattern is to register a pool of buffers once at startup and reuse them for the life of the connection, so the fast path is pure post-and-poll with no per-message setup. This is the RDMA analogue of any connection-pool or buffer-pool optimization: amortize the expensive setup so the hot path is cheap.
Consider congestion beginning to build. Several senders converge on one destination, and a switch on the path starts to fill its buffers. On a lossy TCP network the switch would eventually drop packets and TCP would back off — ugly but self-correcting. On a RoCEv2 fabric, dropping is what we are trying to avoid, so instead ECN kicks in first: the switch marks packets to tell the senders 'you are causing congestion', and well-behaved RDMA congestion control responds by slowing those flows down before the queues overflow. ECN is the gentle, proactive signal, and a well-tuned fabric leans on it to keep flows in check without ever needing the blunt instrument.
Now the blunt instrument, and the failure it can cause. Congestion builds faster than ECN can throttle it, and a switch's buffer for a priority fills to the threshold where it must act to avoid dropping. It sends a PFC pause frame upstream, telling the previous hop to stop sending that priority's traffic. That upstream device, now unable to drain its own buffer, may fill and pause its upstream neighbor — and so the pause propagates backward through the fabric, a phenomenon called congestion spreading, where a hot spot at one destination pauses traffic on links that have nothing to do with it. In the worst case, a cycle of dependencies in the topology causes PFC deadlock: a set of switches all paused waiting on each other, and traffic stops. This is the characteristic serious RDMA incident, and it is why the operational focus is on the fabric: you watch PFC pause counts and ECN marks as leading indicators, tune congestion control so ECN does the work and PFC is a rare last resort, and design the topology and buffer thresholds to keep pausing local. The RDMA endpoints are fast and boring; the fabric's flow control is where RDMA at scale is won or lost, and the teams who succeed with it treat the lossless network as a system to be operated, not a wire to be assumed.