P2P KV Cache Transfer

What actually crosses the wire

The object in flight is the key and value tensors for every prompt token, at every layer. Nothing else moves: the weights are already resident on both sides and the sampled token is a few bytes. The transfer is a bulk copy of one scattered data structure out of the prefill GPU’s HBM into the decode GPU’s.

What makes it interesting is that this copy sits on the critical path of the first token. In a monolithic server the KV cache is produced and consumed in the same memory, so its size is a capacity problem, never a latency one. Disaggregation converts that into a bandwidth problem: can these bytes traverse this fabric inside the TTFT allowance, and how much of that time hides behind work that has not finished yet?

Bytes per token — the arithmetic that sets the budget

The size of a request’s KV cache is fully determined by the model shape and the prompt length. Per token:

bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × dtype_bytes

request_bytes  = bytes_per_token × prompt_tokens

The leading 2 is keys plus values. The term that decides whether disaggregation is affordable is n_kv_heads, not the query head count: under grouped-query or multi-query attention many query heads share one KV head, so the cache, and the transfer, shrinks by that grouping factor. Writing the formula with the full hidden size silently assumes classic multi-head attention and inflates every number downstream several-fold.

As an illustration only: 80 layers, 8 KV heads, head dimension 128, FP16 gives 2 × 80 × 8 × 128 × 2 ≈ 328 KB per token, so a 4,000-token prompt is roughly 1.3 GB. The result is linear in prompt length and lands in the hundreds-of-megabytes-to-gigabytes range.

Layer-wise streaming — start sending before prefill finishes

A naive implementation waits for prefill to complete, then copies the whole cache, serialising two expensive phases. The fix rests on one fact: layer 0’s KV is final the moment layer 0’s attention has run, and nothing later in the forward pass modifies it. So the transfer can begin immediately and proceed layer by layer while the remaining layers still compute.

Implemented with a separate CUDA stream and per-layer events, this turns the cost from prefill + transfer into roughly:

total ≈ max(prefill_compute, total_transfer) + transfer_of_last_layer

That trailing term is the part people forget. The last layer’s KV cannot exist before the last layer runs, so its share of the bytes is structurally unhidable however fast the fabric is. With 80 layers that is roughly 1/80th of the transfer left exposed: small, but a floor, and the reason deep models with many thin layers overlap better than shallow ones with fat layers. See CUDA streams for the overlap mechanics.

Scatter-gather, not one big memcpy

A serving engine does not store a request’s KV as one contiguous buffer, but as fixed-size blocks scattered across a preallocated pool and indexed by a block table — the arrangement paged KV cache exists to explain. That makes the transfer a scatter-gather: many descriptors, one per block per layer, rather than a single large copy.

This matters because descriptor count, not just total bytes, determines whether you reach line rate. Every fabric has a message size below which per-message overhead dominates and achieved bandwidth collapses under the link’s rating. A small block on a model with a modest per-token, per-layer footprint can be tens of kilobytes, small enough that an RDMA engine spends more time on work requests than on payload.

The mitigations all make messages bigger: coalescing adjacent blocks into one descriptor, batching a layer’s blocks into a single scatter-gather list, or choosing a larger block size where you disaggregate.

Three transport paths and what each buys

Where the two GPUs sit relative to each other decides the mechanism, and the mechanisms differ by more than an order of magnitude.

Order-of-magnitude intuition, as arithmetic rather than a spec table: a 400 Gb/s-class NIC is about 50 GB/s of theoretical one-way payload; an intra-node NVLink domain is substantially faster still. The fabrics are covered in NVLink and NVSwitch, InfiniBand and RDMA, and PCIe, P2P, and NUMA. The practical rule: the host-staged path is a correctness fallback, not a deployment target.

Pre-registered memory and true zero copy

RDMA cannot read arbitrary memory. The NIC needs a memory region: pages pinned so they cannot move, with translations the adapter can use directly. Registration is expensive for a large region, and per-request registration would swamp the transfer it enables.

The resolution is that the KV pool is allocated once at engine startup and registered once, in full. Every subsequent transfer addresses into that already-registered region using offsets from the block table, so registration is amortised over the life of the process. This is why a preallocated paged pool is a prerequisite for fast disaggregation, not merely a memory-efficiency trick.

With the region registered on both sides the data path is genuinely zero-copy: the NIC DMAs out of source HBM into destination HBM with no staging buffer and no CPU touching payload. The CPU only posts work requests and reaps completions: control plane, never data plane.

Fitting the transfer inside the TTFT allowance

Turn the bytes into milliseconds and compare against your latency service level objective. The exposed cost is the part streaming cannot hide:

transfer_ms      = request_bytes / achieved_bandwidth
exposed_ms       ≈ max(0, transfer_ms − prefill_ms) + transfer_ms / n_layers
TTFT             ≈ queue + prefill + exposed_ms + first_decode_step

Note the shape of that middle line. If the fabric moves the cache faster than prefill produces it, the only exposed cost is the last layer’s slice plus the handshake, usually invisible. If the fabric is slower, the excess lands directly on every user’s first token.

Use achieved bandwidth, not the link rating: after protocol overhead, descriptor-size effects, and contention from other in-flight requests on the same NIC, a realistic figure is a fraction of nameplate. The fabric is shared, so p99 transfer time under a busy prefill pool is what decides p99 TTFT.

The control plane — allocate, then stream

Before a byte moves, the receiver must have somewhere to put it. The decode worker reserves destination blocks and returns their handles to the sender; only then can the sender post transfers. That allocate-then-stream handshake is a real round trip on the critical path, which is why implementations overlap it with the start of prefill.

Whether the sender pushes or the receiver pulls is a design axis, not a standard. A push keeps the source in control and pairs naturally with layer-wise streaming, since the sender knows the instant each layer is ready. A pull lets the receiver pace ingress and admit work only when it has capacity, at the cost of a notification per layer.

Transport choice follows. Collective libraries assume a communicator across a fixed group, awkward when the two pools scale independently — hence point-to-point engines on UCX-style primitives for this path, while NCCL collectives stay inside each pool for tensor parallelism.

When the receiving pool has no free blocks

The decode pool will run out of blocks: it is a bounded resource and admission is driven by traffic. There are three honest responses and one dishonest one.

Back-pressure is the default: the allocation fails or blocks, the scheduler holds the request, and the prefill GPU keeps the finished cache pinned in its own memory until space opens. That converts a decode-side shortage into prefill-side memory pressure, so it needs a timeout after which the cache is discarded and recomputed. Spilling the incoming cache to host memory trades the shortage for a later, slower reload. Preempting an in-progress sequence frees blocks immediately at the cost of recomputing that victim’s prefill.

The dishonest option is streaming optimistically and discovering mid-flight that the destination is full: fabric bandwidth burned on a transfer you must abort, and partial writes to reclaim. Allocate first, always.

The crossover — when the wire costs more than the split saves

There is a persistent intuition that long prompts amortise the transfer better. The arithmetic dissolves it: prefill compute scales roughly as 2 × params × tokens and KV bytes as bytes_per_token × tokens, so prompt length appears in both and largely cancels in the ratio.

What actually moves the crossover is elsewhere. Fixed overhead (allocation handshake, descriptor setup, completion notification) does not scale with tokens, so short prompts suffer most: a 200-token request pays it against a prefill of a few milliseconds, where it lands as a visible fraction rather than a rounding error. Fabric class dominates everything; the workload that is free over NVLink is ruinous host-staged. And the quadratic attention term helps very long prompts, where compute grows faster than bytes.

So the test is empirical and per-deployment: measure exposed transfer milliseconds against the TTFT budget at your real prompt-length distribution. If the exposed cost approaches the prefill it was meant to isolate, keep prefill and decode colocated.