Why it matters

CPU offload extends effective memory. Understanding shapes design.

Advertisement

The architecture

Identify what to offload (optim + activations + inference KV).

PCIe transfer overhead.

Prefetch overlap.

CPU offload math flowIdentify offload targetsoptim + KV + actsTransfer via PCIebandwidth-boundPrefetch + overlaphide latencyGB200 uses NVLink C2C for CPU coherent memory; PCIe elsewhere
CPU offload math.
Advertisement

How it works end to end

CPU RAM 5-10x GPU HBM.

PCIe 5.0 ~64 GB/s.

NVLink C2C (GB200) 900 GB/s.

Prefetch critical.