Why architecture matters here
Start with the number that dominates GPU performance: memory latency. A global-memory load on a modern GPU costs on the order of 400–800 cycles. If an SM had only one warp resident and that warp issued a load, the SM would sit idle for hundreds of cycles waiting — its arithmetic units, capable of thousands of operations in that window, doing nothing. The GPU's answer is to keep dozens of warps resident and let the scheduler pick a ready one every cycle. With enough warps in flight, the SM always has an instruction to issue while others wait on memory, and the expensive latency disappears behind useful work. Occupancy quantifies exactly how much of this latency-hiding capacity a kernel actually uses.
Why can't you just always run the maximum warps? Because the resources that make a warp resident are finite and shared. Every thread's live variables occupy registers from a fixed per-SM register file; a warp is 32 threads, so a kernel using 64 registers per thread consumes 2,048 registers per warp, and an SM with 65,536 registers can hold at most 32 such warps regardless of any other limit. Every block that uses shared memory reserves it from a fixed per-SM pool, capping how many blocks — and therefore warps — fit. On top of these, hardware imposes hard caps: a maximum number of resident blocks and warps per SM. Occupancy is the minimum of what all these limits allow.
Getting occupancy right delivers three things. Latency tolerance: enough warps to cover memory and pipeline stalls so the SM stays busy, which for memory-bound kernels is the whole game. Throughput scaling: on bandwidth- or compute-bound kernels, adequate occupancy ensures the SM's issue slots and load/store units are saturated. Predictable tuning: knowing which resource limits occupancy tells you exactly what to change — fewer registers, less shared memory, a different block size — instead of guessing.
But occupancy has a ceiling of usefulness and a real cost. Past the point where latency is hidden, extra warps do nothing; and raising occupancy usually means giving each thread fewer registers, which can force the compiler to spill live variables to slow local memory — trading a latency-hiding win for a latency-adding loss. High occupancy also means more threads sharing the L1/L2 caches, which can raise miss rates. The craft is finding the occupancy that hides latency without starving each thread of the registers it needs to do real work — which is why understanding the mechanism beats chasing a percentage.
The architecture: every piece explained
Top row: the four inputs. Threads per block is chosen at kernel launch and, divided by the warp size of 32, sets warps per block. Registers per thread is decided by the compiler based on the kernel's live variables (and influenced by optimization flags and launch bounds); more registers per thread means fewer warps fit in the register file. Shared memory per block is what the programmer allocates for cooperative data; more shared memory per block means fewer blocks fit in the shared-memory pool. Blocks-per-SM and warps-per-SM caps are fixed hardware limits that no amount of tuning can exceed.
Middle row: the binding constraint. The engine computes, for each resource, how many warps that resource would allow: the register file allows register_file / (registers_per_thread × 32) warps; the shared-memory pool allows smem_pool / smem_per_block blocks worth of warps; the hardware caps allow their fixed maxima. The limiting resource is the minimum of these — the scarcest one — and it sets the number of resident warps. This is why tuning the wrong resource is wasted effort: if registers are the limit, halving shared memory changes nothing until registers are relieved.
Bottom row: what resident warps buy. The warp scheduler on each SM holds the resident warps and, every issue cycle, selects one that is ready (not stalled on a dependency or memory). Switching between warps is free — no context save/restore, because every warp's state already lives in the register file and shared memory. This zero-cost switching is what enables latency hiding: while warp A waits on a global load, warps B, C, and D issue arithmetic, so the memory stall is overlapped with compute rather than exposed as idle time. More resident warps means more candidates to hide behind — up to the point where latency is fully covered.
Bottom strip: the operational surface. The occupancy calculator (and the compiler's --ptxas-options=-v register/smem report) predicts theoretical occupancy from a kernel's resource usage before you run it. Achieved occupancy, measured by profilers like Nsight Compute, reveals the runtime reality — often lower than theoretical because of tail effects, uneven block completion, or launch configurations that don't tile evenly onto SMs. Launch bounds let you cap registers to guarantee a target occupancy. And the roofline model tells you whether the kernel is even latency-bound — the only regime where raising occupancy helps. The diagram shows the four inputs feeding a min() that sets resident warps, which feed the scheduler that hides latency.
End-to-end flow
Walk a concrete kernel onto an SM. Suppose the SM has a 65,536-entry register file, 100 KB of shared memory, a cap of 32 resident warps, and a cap of 16 resident blocks. You launch a kernel with 256 threads per block (8 warps per block), and the compiler reports 40 registers per thread and 8 KB of shared memory per block. Now compute each resource's warp allowance.
Registers: 40 registers/thread × 32 threads/warp = 1,280 registers per warp; 65,536 / 1,280 = 51 warps the register file could hold — but rounded to whole blocks of 8 warps, that is 6 blocks (48 warps). Shared memory: 100 KB / 8 KB = 12 blocks. Block cap: 16 blocks. Warp cap: 32 warps = 4 blocks. The minimum is the warp cap at 4 blocks (32 warps) — here the hardware warp limit binds first, so theoretical occupancy is 32/32 = 100%. Registers and shared memory have slack; tuning them would not help because they are not the constraint.
Now change one thing: the kernel grows to need 64 registers per thread (a more complex inner loop). Registers per warp become 64 × 32 = 2,048; 65,536 / 2,048 = 32 warps, but in whole 8-warp blocks that is 4 blocks (32 warps) — still fine. Push to 96 registers: 96 × 32 = 3,072 per warp; 65,536 / 3,072 = 21 warps → 2 blocks (16 warps). Occupancy just fell to 16/32 = 50%, and registers are now the limiting resource. If the kernel was memory-latency-bound and 16 warps no longer hide the latency, throughput drops — this is an occupancy cliff caused by register pressure.
At runtime the scheduler makes occupancy pay off. With 32 resident warps, warp 0 issues a global load and stalls for ~500 cycles; the scheduler immediately issues from warps 1–31, which are doing arithmetic on data already in registers. By the time warp 0's data arrives, the SM never idled — the 500-cycle latency was fully overlapped. Drop to 4 resident warps and the same load leaves the SM with too few ready warps to fill 500 cycles; the pipeline drains and the SM stalls. The profiler would show high achieved occupancy and low stall in the first case, and low occupancy with high memory-dependency stall in the second — the exact signal that says 'this kernel needs more warps, so cut register usage or add launch bounds.' Conversely, if the profiler shows the SM already saturated at 4 warps (a compute-bound kernel with good ILP), raising occupancy would buy nothing, and the right move is to leave registers alone.