Why architecture matters here
CUDA streams and graphs matter because they're essential to GPU performance -- streams for keeping the GPU busy via overlap, graphs for eliminating launch overhead -- and getting them right is often what separates good GPU utilization from poor. A GPU is fast, but achieving its potential requires feeding it work continuously (never idle -- via overlap of independent operations, which streams enable) and submitting work efficiently (minimal launch overhead -- which graphs address). Without streams (all work serialized), the GPU waits (e.g., idle during data copies -- the copy and compute serialized, not overlapped); without graphs (each kernel launched individually), the launch overhead can dominate (for many-small-kernel workloads, the CPU spends more time launching than the GPU spends computing -- the CPU can't keep up). Both mechanisms are crucial for GPU-intensive workloads (deep learning training and inference, HPC) -- streams for overlap (utilization), graphs for launch efficiency (reducing overhead). For anyone optimizing GPU performance, understanding streams (overlap) and graphs (launch overhead) is essential, and they're increasingly important as workloads (like LLM inference) have many small kernels where launch overhead matters.
The overlap insight (streams) is a key performance concept, and it's about keeping the GPU busy. A GPU operation (a kernel, a data copy) takes time, and if operations are serialized (one after another), the GPU may be idle during some (e.g., during a data copy, the compute units are idle -- waiting for the data). Streams enable overlap: operations in different streams can execute concurrently, so independent operations overlap (e.g., while stream A computes on already-loaded data, stream B copies the next batch of data -- the copy hidden behind the compute, so the GPU's compute units stay busy and the data path is used concurrently). This copy-compute overlap is a classic pattern (overlapping data transfer with computation -- so the data transfer is hidden, not adding to the total time) -- keeping the GPU busy (computing) while data moves. More generally, streams enable concurrency (independent work overlapping -- multiple streams' work interleaved on the GPU) -- maximizing utilization (the GPU always has work, never idle waiting). Understanding that streams enable overlap (concurrent independent operations -- hiding latencies like data transfer, keeping the GPU busy) is understanding how to achieve high GPU utilization, and it's why streams are fundamental to GPU performance.
And the launch-overhead-elimination insight (graphs) is the other key concept, increasingly important for many-small-kernel workloads. Each CUDA kernel launch has a CPU-side cost (the CPU prepares the launch -- setting up parameters, submitting to the GPU -- taking some microseconds per launch). For workloads with a few large kernels, this is negligible (the launch overhead tiny relative to the kernel compute). But for workloads with many small kernels (like LLM decode -- generating each token involves many small kernels, and each token is fast), the launch overhead becomes significant (the per-launch overhead, multiplied by many kernels, can rival or exceed the compute time -- and worse, the CPU may not be able to submit launches fast enough to keep the GPU fed, so the GPU idles waiting for the CPU to launch the next kernel -- CPU-bound, not GPU-bound). CUDA graphs eliminate this: capture the sequence of kernels once (recording the work -- the kernels and their dependencies), then replay the whole graph cheaply (launching the entire captured sequence with a single, low-overhead operation -- the GPU executes the recorded sequence without per-kernel CPU launch overhead). So for the repeated sequence (e.g., the decode step, repeated per token), capture it once and replay -- eliminating the per-launch overhead (the graph replay having minimal overhead regardless of the number of kernels in it). This -- capturing a repeated kernel sequence once and replaying it cheaply, eliminating per-launch overhead -- is crucial for many-small-kernel workloads (LLM decode being the prime example), and understanding that graphs eliminate launch overhead (by replaying a captured sequence) is understanding how to make such workloads efficient.
The architecture: every piece explained
Top row: streams and graphs. Streams: ordered queues of GPU work -- operations in a stream execute in order, operations in different streams can execute concurrently -- the concurrency mechanism. Concurrency: overlapping independent operations (compute in one stream, copy in another -- concurrent) -- keeping the GPU busy (utilization). Events and sync: cross-stream synchronization -- events (recorded in a stream, waited on by another -- expressing dependencies: stream B waits for stream A's event before proceeding) -- coordinating work across streams. CUDA graphs: captured work replayed -- a sequence of operations captured once (recorded) and replayed cheaply (launched with minimal overhead) -- eliminating per-launch overhead for the repeated sequence.
Middle row: the graph mechanics and overlap. Launch overhead: the per-kernel CPU cost of launching (preparing and submitting the launch) -- negligible for few large kernels, significant for many small kernels (dominating, or the CPU unable to keep the GPU fed). Graph capture: recording a sequence of operations once (capturing the kernels and their dependencies into a graph) -- the one-time recording. Graph replay: launching the captured graph cheaply (executing the recorded sequence with minimal overhead -- regardless of the number of kernels) -- the cheap repeated execution (eliminating the per-launch overhead). Copy-compute overlap: overlapping data transfer (copy) with computation (via streams -- copy in one stream, compute in another) -- hiding the data transfer behind computation (so it doesn't add to the total time).
Bottom rows: pitfalls and application. Default stream pitfalls: the default stream has implicit synchronization semantics (depending on the configuration -- the legacy default stream synchronizes with all other streams, serializing work unexpectedly) -- a common pitfall (work unexpectedly serialized because it's on the default stream); using explicit non-default streams (or the per-thread default stream) avoids it. LLM inference use: CUDA graphs are heavily used in LLM inference (the decode step -- many small kernels per token -- captured as a graph and replayed per token, eliminating the launch overhead that would otherwise dominate or bottleneck the CPU) -- a prime application. The ops strip: stream design (designing the stream structure -- which work in which streams, the dependencies via events -- for overlap and correctness), graph capture (capturing the repeated sequences as graphs -- identifying the repeated work, capturing and replaying it for launch-overhead elimination), and profiling (profiling the GPU work -- checking utilization, overlap, launch overhead -- via tools like Nsight -- to identify where streams/graphs help).
End-to-end flow
Trace copy-compute overlap with streams. A workload processes batches of data on the GPU: each batch is copied from host to device, then computed on. Without streams (serialized): copy batch 1, compute batch 1, copy batch 2, compute batch 2, ... -- the GPU idle during each copy (the compute units waiting for the data), and the data path idle during each compute -- so the total time is copy + compute + copy + compute + ... (serialized). With streams (overlapped): using two streams, while stream A computes batch 1, stream B copies batch 2 (concurrently -- the copy of batch 2 overlapping the compute of batch 1) -- so the copies are hidden behind the computes (the data transfer overlapping the computation) -- the total time approaching just the compute time (the copies hidden) -- much faster. The streams enabled the copy-compute overlap (concurrent copy and compute in different streams), keeping the GPU busy (computing) while data moves -- the overlap improving utilization and throughput.
The launch-overhead and graph vignettes show the graph benefit. A launch-overhead case: an LLM decode step (generating a token) involves many small kernels (the attention, the FFN, the various operations -- many small kernels, each fast). Launching each kernel individually, the per-launch CPU overhead (microseconds per launch, times many kernels per token, times many tokens) becomes significant -- and worse, the CPU can't submit the launches fast enough to keep the GPU fed (the GPU idles waiting for the CPU to launch the next kernel -- CPU-bound, the GPU underutilized). The team uses a CUDA graph: they capture the decode step (the sequence of kernels for generating a token) once as a graph, then replay it per token -- so each token's generation is a single cheap graph replay (the whole sequence of kernels launched with minimal overhead) instead of many individual launches -- eliminating the per-launch overhead (the GPU no longer waiting for the CPU to launch each kernel -- the graph replay feeding the GPU efficiently). The decode throughput improves (the launch overhead eliminated, the GPU kept fed) -- the CUDA graph addressing the many-small-kernel launch overhead that dominated.
The default-stream and profiling vignettes complete it. A default-stream case: the team notices their overlapping work isn't overlapping (the copies and computes serialized despite using streams) -- investigating, they find some work is on the default stream (which has implicit synchronization -- the legacy default stream synchronizing with the other streams, serializing the work). The fix: use explicit non-default streams for all the work (avoiding the default stream's implicit synchronization) -- so the streams overlap correctly (the copies and computes concurrent). The default-stream pitfall (implicit synchronization serializing work) was avoided by using explicit streams. A profiling case: the team profiles the GPU work (via Nsight -- examining the timeline of streams, kernels, copies) -- confirming the overlap (copies hidden behind computes) and the graph replay (the decode as cheap graph replays, low launch overhead) -- and identifying further optimization opportunities (any remaining serialization or overhead) -- the profiling guiding the stream/graph optimization. The consolidated discipline the team documents: use streams for overlap (concurrent independent operations -- copy-compute overlap hiding data transfer, keeping the GPU busy -- utilization), synchronize across streams correctly (events for dependencies), use CUDA graphs to eliminate launch overhead (capturing repeated sequences -- like LLM decode -- once and replaying cheaply, crucial for many-small-kernel workloads), avoid the default-stream pitfalls (use explicit non-default streams to prevent implicit serialization), design the stream structure for overlap and correctness, and profile the GPU work (identifying where streams/graphs help) -- because CUDA streams (overlap, keeping the GPU busy) and graphs (eliminating launch overhead for many-small-kernel workloads) are essential to GPU performance, and using them correctly (overlap via streams, launch-overhead elimination via graphs, avoiding the default-stream pitfalls) is often what separates high GPU utilization from poor.