Most advice about GPU profiling skips the part that actually decides whether your session is useful: what the tool records, and what it does to your program while recording it. Nsight Systems and Nsight Compute are not two views of one dataset. They acquire different things by different means, distort your workload in different ways, and disagree about something as basic as how long a kernel takes — for good reasons. This article is about the instruments: which rows the timeline draws, how to capture a window out of a server that never exits, why Nsight Compute runs your kernel more than once, and how to read the report without inventing a story it does not support.
Two tools, two different recordings
Nsight Systems is a tracer. It sits between your process and the CUDA driver, timestamps events — API calls, kernel executions, copies, NVTX ranges — and writes them to a report with a common clock. It does not know why anything was slow; it knows what happened and when, across every thread and every GPU the process touched.
Nsight Compute is a counter collector. It takes over one kernel launch at a time, programs the GPU’s hardware performance counters, and reports what those counters saw. It has no timeline at all.
That difference in mechanism is why you run them separately, why their kernel durations are not comparable, and why the order is Systems first to find which kernel deserves attention, Compute second.
What the Nsight Systems timeline rows actually are
A report is a tree of rows, and reading it starts with knowing which layer each row belongs to. Under the process you get CPU thread rows and, nested under them, the CUDA API row: host-side calls as the calling thread experienced them — cudaLaunchKernel, cudaMemcpyAsync, cudaStreamSynchronize — each bar spanning the call’s entry and return.
Separately, grouped per GPU and per CUDA context, are the device rows: a kernel row and memory rows (host-to-device, device-to-host, device-to-device, memset) per stream. Optional traces add rows for cuBLAS, cuDNN and NCCL, and NVTX gets rows of its own.
The mechanic that trips people up: the API row and the kernel row are different events. A short cudaLaunchKernel bar is the cost of enqueuing work; the kernel bar below it is the execution, and it starts later.
Correlation, and how to read a gap
Every traced API call carries a correlation ID linking it to the device activity it produced — that is what the UI draws when you select a kernel and it highlights the launch that submitted it, and it is the only reliable way to answer “which host call put this bar here.”
A gap is just an interval with no recorded activity on a row, and reading one is a matter of asking which other rows were occupied during it. A quiet kernel row on stream A while stream B is busy is concurrency. A quiet kernel row under a long cudaStreamSynchronize bar on the API row is the host waiting. A gap with nothing recorded anywhere may simply mean the responsible subsystem was not traced. Note the discipline: the tool bounds the gap in time and says which rows were busy. Naming a bottleneck is a separate exercise.
NVTX — making the timeline legible
Out of the box, a timeline is thousands of anonymous kernel bars. NVTX is the annotation ABI that fixes this: your code pushes and pops named ranges, emits instantaneous marks, and names resources (threads, CUDA streams, devices) so the rows arrive labeled instead of numbered. Nsight Systems records these into their own rows, nested by push depth, aligned to the same clock as everything else.
Annotate the units you actually reason in — a request, a batch, a prefill, one decode step, a layer — not every function. Bindings exist for C/C++ and most frameworks.
Two cautions. Ranges are cheap but not free, so annotating something invoked hundreds of thousands of times per second is itself a perturbation. And keep names low cardinality: a unique string per request bloats the report’s string table and defeats grouping.
Capturing against a long-running server
The default workflow — launch under the profiler, let it exit, open the report — is built for a script. A serving process never exits, and left running it produces a report too large to open. Four mechanisms carve out a window instead:
Delay the start so you skip process startup, weight loading, autotuning and graph capture. Duration-limit the capture so it stops itself after a fixed number of seconds. Bound the range explicitly, either with the cudaProfilerStart / cudaProfilerStop API or by nominating an NVTX range as the capture trigger — the cleanest option, because the window then means something in your domain. And you can attach a capture session to an already-running process.
Combine that with restricting which subsystems you trace. The workable recipe: warm the server, drive it with the real load generator, capture a few seconds of steady state, stop cleanly.
Why Nsight Compute replays your kernel
The GPU has a finite number of counter collection units, and a full set of sections asks for more counters than can be programmed at once. So Nsight Compute runs the kernel several times, a different group of counters each pass.
Under the default kernel replay, the tool snapshots the memory the kernel may write before the first pass and restores it before each subsequent pass, so every pass sees identical inputs. Application replay re-runs the whole application once per pass instead, which suits kernels whose state cannot be captured that way but demands a deterministic program. Range replay replays a bounded sequence of API calls, for kernels that only make sense in the context of the ones around them.
The consequences are practical: profiling one kernel costs many executions of it, non-deterministic kernels can yield inconsistent numbers, and you almost always want to filter to specific kernel names or launch indices rather than profiling every launch.
What the Nsight Compute sections report
The report is organized into sections, each a different family of counters. GPU Speed Of Light Throughput gives the top-level achieved fractions of peak for the compute and memory paths. Memory Workload Analysis draws the hierarchy — requests and sectors moving between the SM, L1/shared, L2 and device memory, with hit rates at each level. Compute Workload Analysis breaks utilization down by pipeline.
Scheduler Statistics reports, per warp scheduler per cycle, how many warps were resident, how many were eligible to issue, and how often one actually issued. Warp State Statistics gives a sampled distribution of the states warps were sitting in while not issuing. Occupancy contrasts achieved with theoretical and names the resource that caps it; Launch Statistics records grid and block shape, registers and shared memory; Source Counters attributes samples to SASS instructions and, with line info compiled in, to source lines. Read warp-state names as labels for what a warp was waiting on — observations, not verdicts.
Overhead, and how it distorts what you measure
Every profiler perturbs its subject, and knowing the shape of the perturbation is part of using the tool. Nsight Systems adds a small fixed cost per traced event. That is negligible for a workload dominated by long kernels and very much not negligible for one issuing an enormous number of tiny API calls, where the traced launch path looks wider than the production one. Tracing fewer subsystems is the direct lever; turning off CPU sampling is another.
Nsight Compute distorts far more, deliberately. It serializes kernel execution, so concurrency between streams disappears from what you measure. It flushes caches between passes by default, so each pass starts cold — that is what makes hit rates reproducible and also what makes them unrepresentative of a hot steady state. It can lock clocks to a fixed base for comparability. The rule that follows: take ratios and counters from Compute, wall-clock from Systems.
Multi-GPU and multi-process capture
One Nsight Systems report covers one process, so a single process driving eight GPUs produces eight groups of device rows on one timeline — ideal, because copies and collectives between them line up on a shared clock.
The common serving and training layout is the opposite: one process per GPU under a launcher. There, profile per rank and put the rank in the output filename, because each process writes its own report. Capturing every rank is usually a mistake — the disk traffic is real and the reports open separately — so take one or two. Profiling the launcher works only if the tool is told to follow the child processes it spawns. Enabling the NCCL trace gives you rows for collectives, which is what makes a rank’s report legible at all.
One environmental gotcha: hardware counter collection is permission-gated by the driver, and containers need that capability granted explicitly. Plain tracing generally is not gated.
Reading a report without jumping to conclusions
Before interpreting anything, validate the capture. Did the window land on steady state or on warmup? Does the report warn about dropped events or buffer overflow — because a saturated trace buffer produces gaps that are artifacts of the tool? Was it truncated mid-request by a duration limit? Are the subsystems you are about to reason about actually in the trace?
Then compare like with like. Nsight Compute can import an earlier report as a baseline and show per-section deltas, which is more trustworthy than one absolute number; two captures of the same build should agree before you believe either.
Finally, resist the pull of the single headline percentage. An achieved fraction of peak is a number produced under this capture’s conditions and this tool’s distortions. The profiler’s contribution is a defensible measurement; deciding what constrains the workload is the next step, not this one.