nvidia-smi — The GPU Command-Line Tool

Why it matters

nvidia-smi is the first tool anyone reaches for. Understanding it is basic GPU literacy - and it repays a careful reading, because almost every number in its default table means something narrower than its label suggests.

The tool itself is a thin command-line front end over NVML, the NVIDIA Management Library. That is the same C API that DCGM, the Kubernetes device plugin, every Prometheus GPU exporter and every cloud console's GPU page call underneath. The practical consequence is worth internalising: anything nvidia-smi can print, a long-lived program can read without forking a process, and anything nvidia-smi cannot print, NVML cannot report either. It is a management interface, not a profiler. It knows the state of the board at a coarse sampling rate and it knows nothing whatsoever about your kernels, your tensors or your model.

That fixes its place in the toolchain. nvidia-smi answers is this board healthy, correctly configured, and doing something. Nsight answers why is this kernel slow. DCGM answers what did five hundred boards do last week. Most of the confusion around GPU operations comes from asking one of those tools the other two questions.

Reading the default table

The header line reports the driver version and a CUDA Version. That second field is the highest CUDA runtime version this driver is capable of supporting - it is not the toolkit you installed and not the version your binary links against. A machine printing CUDA Version: 12.4 with a 12.1 toolkit on disk is normal and healthy. Misreading that one line is behind a remarkable amount of wasted debugging, and it is worth checking nvcc --version before concluding anything about a mismatch.

Each board then occupies two rows. The upper row carries the index, the product name, persistence mode, the PCI bus id, whether a display is attached, and the volatile uncorrectable ECC count. The lower row carries fan speed, temperature, performance state, power draw against the enforced cap, memory used against total, GPU utilization, compute mode and MIG mode.

Three of those fields are read wrongly often enough to call out. The performance state runs from P0 (maximum clocks) down through P8 or P12 at idle; a board pinned at P0 with zero utilization is not busy, it is one that has been told to stay awake by persistence mode or an application clock setting. The bus id is the PCI address, and it is the only field in that row that maps to a physical slot - the leading index does not, for reasons developed below. And Volatile Uncorr. ECC is the single field in the table whose non-zero value should wake somebody: it counts uncorrectable memory errors since the driver last loaded, and uncorrectable means the data was wrong and nothing fixed it.

The display-attached column is a smaller signal in the same family. On a datacenter board it should read Off; an unexpected On usually means something claimed the device through a graphics path you did not intend, which costs you memory and, on some platforms, the ability to reset the board cleanly.

GPU-Util is a duty cycle, not a busy-ness meter

The utilization percentage is the fraction of the last sampling window during which at least one kernel was resident on the device. That is the whole definition. It is not the fraction of SMs doing work, not the fraction of issue slots filled, and not occupancy.

Work through what that permits. A kernel launched with a single block, occupying one SM out of a hundred and something, running back to back, reports 100%. A meticulously tuned kernel that saturates every tensor core on the die but only runs during 40% of wall clock reports 40%. The number tells you the device was not idle. It tells you nothing about whether the device was well used, and the gap between those two statements is where a lot of capacity gets bought unnecessarily. "We are at 95% GPU utilization, we need more GPUs" is a claim that needs a second measurement before it becomes a purchase order.

The second measurement depends on the question. How many warps are actually resident and what is stopping more of them from being resident is occupancy, developed in GPU occupancy. Which kernel, for how long, and stalled on what is a profiler question and belongs to Nsight Systems and Nsight Compute.

The sampling window is its own trap. The driver measures over a short interval, so a workload made of many short kernels with gaps between them can read anywhere across a wide band depending on where the interval boundaries happen to fall. Single readings jitter; a screenshot proves nothing. Average a loop over tens of seconds before you believe a utilization figure, and be suspicious of dashboards that scrape once a minute and plot the instantaneous value.

One more field shares the confusion. utilization.memory is the same style of duty cycle applied to the memory controller - the fraction of time it was reading or writing - and not the fraction of capacity in use. Two fields with "memory" in the name, two entirely different meanings.

What Memory-Usage actually counts

The memory column reports what the driver has handed out on this device. That is strictly more than your tensors and considerably less informative than people assume.

Start with the fixed cost. Every process holding a CUDA context pays for that context before it allocates a single byte of its own - a few hundred megabytes, growing with the number of kernels loaded into the context and again when libraries such as cuBLAS or cuDNN allocate their workspaces on first use. Eight worker processes sharing one board pay it eight times, which is why a card can show several gigabytes in use with every model still on disk.

Then the allocator. Frameworks with caching allocators request large slabs from the driver and, by design, do not return them when a tensor is freed. nvidia-smi sees the slabs. A job whose live tensors total twenty gigabytes can legitimately show sixty, and nothing is wrong. If the question is "is this job close to running out of memory", the framework's own accounting - its allocated figure against its reserved figure - is the number that answers it. If the question is "can another job fit on this board", nvidia-smi is right and the framework's figure is the misleading one, because those cached slabs really are unavailable to anyone else.

Two smaller effects round it out. Total memory can be below the datasheet figure when error protection carves check bits out of the array rather than using dedicated storage - what that costs is a memory-architecture question covered in GPU HBM. And fragmentation is completely invisible here: a board reporting thirty gigabytes free can still fail a four-gigabyte contiguous allocation, and no field in this tool will hint at why.

Query mode - the machine-readable interface

The default table is formatted for a human at a terminal. Parsing it with awk is a mistake that survives exactly until the next driver release changes a column width or adds a field. The supported machine interface is the query mode, which emits one record per device with the fields you asked for and nothing else.

# one line per GPU, no header, bare numbers - the shape a metrics pipeline wants
nvidia-smi --query-gpu=index,uuid,name,memory.used,memory.total,\
utilization.gpu,temperature.gpu,power.draw,clocks.sm \
--format=csv,noheader,nounits

# what is running on the device and how much each process holds
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv

# everything the driver will report, labelled and grouped
nvidia-smi -q
nvidia-smi -q -d PERFORMANCE,ECC,CLOCK,MEMORY

# repeat the query every five seconds
nvidia-smi --query-gpu=index,utilization.gpu,memory.used --format=csv -l 5

# which field names does THIS driver actually accept?
nvidia-smi --help-query-gpu

That last command matters more than it looks. Field names have been added, renamed and deprecated across driver generations, so a query list copied from a blog post can fail on a different node in the same fleet. Ask the driver in front of you which fields it supports and build the list from that, rather than shipping a hardcoded string and discovering the gap during an incident.

Two formatting habits are worth fixing early. Pass nounits for anything a program will parse, so you get 81559 rather than 81559 MiB and no downstream string surgery. And key every record on uuid, never on index - the reason is the next section.

There is also a cost worth knowing before you schedule it. Each invocation initializes NVML, reads, and exits. On one workstation at five-second intervals nobody notices. Across a few thousand boards at one-second intervals you have built a fleet-wide fork storm, and that is precisely the point at which a long-lived NVML client is the correct shape rather than a cron job around this binary.

The index nvidia-smi prints is not the index CUDA uses

nvidia-smi enumerates devices in PCI bus order. The CUDA runtime, by default, does not - it orders devices by a capability heuristic that puts the most capable board first. On a homogeneous node the two orderings usually coincide and nobody notices. On a node with mixed boards they diverge, and GPU 0 in the table is then a different piece of silicon from cuda:0 in your program. Setting CUDA_DEVICE_ORDER=PCI_BUS_ID forces the runtime to agree with the tool, and on any machine you will debug more than once it should simply be set.

A second renumbering sits on top. CUDA_VISIBLE_DEVICES=2,3 makes your process see two devices numbered 0 and 1 that are physically 2 and 3. NVML is unaffected by that variable, so nvidia-smi in the same shell still shows the full machine with the original numbering. Container runtimes add a third layer by exposing only a subset of device nodes, so nvidia-smi run inside the container sees a renumbered subset that matches neither the host table nor necessarily your framework's view.

Three numbering schemes for the same hardware is the situation, and the way out is not to reconcile them but to stop using numbers. The GPU UUID and the PCI bus id are stable across processes, containers, reboots and driver reloads. Log both wherever you record a GPU event. An incident timeline written in terms of "GPU 3" is worth nothing the moment it crosses a process boundary; one written in UUIDs still resolves a year later when you are deciding whether a board has a history.

The topology matrix from nvidia-smi topo -m lives in the same family of stable identifiers, and reading it - what the connection legend means and how to use it for placement - is developed in PCIe host interconnect and GPUDirect rather than repeated here.

Settings that change the node, not your process

A group of flags do not read state, they write it. What they write outlives the process that set it, needs root, and in several cases needs the device idle or reset. Treat all of them as node configuration owned by whatever provisions the machine - not as something a training job flips on its way in, because the next tenant inherits it.

Persistence mode. When no client holds a device open, the driver tears down its state for that device. The next process to touch it pays full device initialization, which on a large-memory datacenter board is seconds, and that cost is billed to whichever request happened to arrive first: a cold first inference, a liveness probe that times out, a benchmark whose first iteration is a wild outlier. nvidia-smi -pm 1 enables the legacy persistence mode; on current drivers the supported mechanism is the nvidia-persistenced daemon, which simply holds a reference to every device so the state never unloads. Either way it does not survive a driver reload, so it belongs in node bring-up rather than in somebody's shell history.

Compute mode. The default permits many processes per GPU. Exclusive-process mode permits one, and a second process receives a clear error instead of quietly competing for memory with the first. That is how you prevent accidental co-tenancy on a training node. It also interacts with every deliberate sharing mechanism, and those have their own treatments: GPU sharing strategies, time slicing and MIG.

Clocks and power. -pl sets the enforced board power limit, and the application-clock and locked-clock flags pin the SM clock instead of letting the boost algorithm choose it. Pinning clocks is the standard move for making benchmarks reproducible, because an unpinned board makes every run a slightly different experiment as it heats up. What capping power actually buys and costs at rack scale is GPU datacenter power's subject.

Throttle reasons. When the reported SM clock sits well below the maximum, nvidia-smi -q -d PERFORMANCE prints the active clock-throttle reasons as labelled lines. The exact spellings have churned across driver generations, so reason about the classes: idle, because there is nothing to run; an applications-clock or user-imposed limit, meaning you asked for this; a software power cap, meaning the board reached its power limit and the driver backed the clocks off; and hardware slowdown, meaning the board asserted an emergency thermal or power signal in silicon. The first three are policy and are usually fine. Hardware slowdown is not policy - it is a cooling or power-delivery failure, and a fleet where it appears on the same racks every afternoon has an airflow problem rather than a GPU problem.

nvidia-smi -r resets a device. It needs nothing running against it, and it is the recovery step that several other operations explicitly demand. On a board wedged badly enough, the reset itself fails and a node reboot is the only remaining option.

ECC state, retired pages and remap availability

Error correction has a current mode and a pending mode, and the difference causes real confusion. Writing nvidia-smi -e 0 or -e 1 sets the pending value; the current value changes only after a GPU reset. A node that appears to have ignored your ECC change has almost certainly just not been reset yet, and a fleet carrying mixed current and pending state is a fleet that will silently change behaviour at the next maintenance reboot. Report both values, not one.

There are two families of counters, and they answer different questions. Volatile counters reset when the driver reloads and tell you what has gone wrong since this node booted. Aggregate counters persist in the board's onboard inforom for the life of the card and tell you whether this board has been degrading for a year - which is the question that matters when deciding whether to keep it in service, and the one you lose forever if you only ever scrape the volatile view.

Correctable single-bit errors are fixed in flight and counted. A slow background trickle across a large fleet is expected. An accelerating rate concentrated on one board is a signal, and it is the earliest one available. Uncorrectable multi-bit errors are not fixed: they poison whatever read that memory, and the usual outcome is a killed process and a device that should not be handed to the next job. The Volatile Uncorr. ECC column in the default table is exactly this counter, which is why it deserves the alert.

What the driver does about persistently bad memory differs by generation, and both mechanisms surface as readable fields. Older boards retire pages: a page that took an uncorrectable error, or that accumulated a pattern of correctable ones, is withdrawn from service, and a retirement that has not taken effect yet shows as pending - meaning the board needs a reset before it is trustworthy. Ampere-class and later boards remap rows from a reserve of spares instead, and the tool reports remap availability as a per-bank histogram of remaining spare capacity together with a remap-failure indicator. Two things there are worth alerting on: a remap failure, which means the board could not repair itself and should be pulled, and a steadily shrinking availability histogram, which is a board telling you months in advance that it is on its way out.

What those check bits cost in capacity and bandwidth is a memory-architecture question and belongs to GPU HBM. The point here is narrower and operational: every one of these fields can be read on a live node without stopping the job that is running on it.

Faults, sampling views, and where nvidia-smi stops

Xid events are not in the table. nvidia-smi -q surfaces error state, but the driver's fault stream is a kernel-log channel: Xid events land in dmesg and the system journal, carrying a number that names a class of fault. The class is what decides the response - an application-level memory fault points at your code, an uncorrectable ECC event points at the board, a link error points at the fabric, and a device that has stopped answering on the bus points at hardware or a reboot. NVIDIA publishes the authoritative table of those numbers and it is the thing to consult rather than a remembered mapping. The operational discipline is simpler than the taxonomy: collect kernel log lines alongside your GPU metrics, because the metric showing that a job died and the line explaining why live in two different places, and correlating them after the fact is much harder than shipping them together.

The sampling views. nvidia-smi pmon prints a rolling per-process sample and nvidia-smi dmon a rolling per-device sample, roughly once per second, one line per interval. They are the closest thing this tool has to a timeline and they genuinely earn their place for questions like "which of the four processes on this board is the one spiking". Two caveats. The per-process activity column carries the same duty-cycle definition as the main utilization field, with all the same limits. And one sample per second is orders of magnitude too coarse to see anything kernel-shaped, which is where the profiler takes over.

MIG. With MIG enabled the mode column flips and a second table lists instances by GPU, instance and compute-instance id with their own memory figures. The per-board utilization column stops meaning anything useful and typically reads as unavailable, because activity is not attributed per instance at this level. That gap - the absence of per-tenant telemetry - is one of the concrete reasons fleets outgrow this tool. Sizing and carving instances is MIG's subject.

Containers. Run inside a container, the tool reports the whole physical board rather than the slice the job was promised, and the process list is frequently blank because the PIDs belong to another namespace; GPU-enabled containers develops that and the sizing bugs it causes.

The boundary. nvidia-smi has no history, no aggregation across nodes, and no access to the hardware profiling counters, and every invocation pays to initialize NVML and then throws that state away. Fleet monitoring wants the exact opposite: a resident client, per-instance attribution, scheduled health checks, and the profiling-derived metrics - tensor pipe activity, DRAM activity, achieved occupancy - that only the profiling interface exposes. That is DCGM's job, and it is the right answer above roughly the point where you stop knowing each node by name. nvidia-smi remains what you run on the one node that is behaving strangely, which is a job it does better than anything else.