Warp Scheduling on GPU

Resident, eligible, selected — three different states

A warp on an SM is always in exactly one of a few states, and conflating them is the single most common misreading of GPU performance data.

Resident (or active) means the warp has been allocated a warp slot and a register range on the SM and will keep them until it exits. How many warps fit is pure resource arithmetic — the occupancy question, treated separately on this site.

Eligible means something much stronger: the warp’s next instruction can issue this cycle. Its source operands are available, the functional unit or issue port it needs is free, it is not parked at a barrier, and no dependency scoreboard entry is outstanding against it.

Selected is the one eligible warp the scheduler actually picks this cycle. Everything else that was eligible is, in counter terms, stalled — which is why the word “stall” on a GPU needs to be read carefully rather than treated as a synonym for trouble.

One cycle in the life of a scheduler

A modern SM is partitioned into several processing blocks, each with its own warp scheduler, its own register file slice, and its own set of resident warps. The schedulers do not share candidate pools: a warp belongs to one partition for its lifetime, so a partition starved of eligible warps cannot borrow from a neighbour. Block sizes that do not divide evenly across partitions can therefore leave some schedulers underfed.

Each cycle the scheduler surveys its warps, computes the eligible set, picks one, and dispatches its instruction to the appropriate pipe. The switch is free: because every resident warp holds its registers for its whole lifetime, there is no context to save or restore — the scheduler just reads a different row. Where a CPU spends transistors on speculation so one thread keeps moving, the SM spends them on register file capacity so that another warp is always ready instead.

The wasted issue slot is the real cost

The right unit of GPU waste is not the stalled warp, it is the empty issue slot. Each scheduler gets one opportunity per cycle to start an instruction. If at least one warp is eligible, that opportunity is used and the SM made forward progress. If zero warps are eligible, that opportunity evaporates — there is no backlog to catch up on later, no queue that drains, just a cycle in which the partition did nothing.

Hence the metric that matters most is issue-slot utilisation: the fraction of active cycles in which a scheduler issued anything at all. A kernel where every scheduler issues most cycles is running well no matter how “stalled” the warp-state histogram looks, because those stalls are warps queueing behind a warp that did issue. A kernel issuing on a small fraction of cycles has a real problem, and the stall taxonomy is how you find out which one. The target is modest: about one eligible warp per scheduler per cycle, and everything past that is insurance rather than throughput.

The scoreboard, and why there are two kinds

Eligibility is enforced by dependency tracking in hardware, and GPUs split it by whether an instruction’s latency is known at compile time.

Fixed-latency instructions — ordinary arithmetic on the math pipes — take a predictable number of cycles, so the compiler encodes the required wait directly into the instruction stream rather than spending a scoreboard entry. A consumer placed too close to its producer cannot issue until that count expires.

Variable-latency instructions — anything that touches memory, plus special-function and some conversion pipes — complete whenever they complete. An L2 hit and an HBM miss differ by hundreds of cycles, so the compiler cannot pre-compute a wait. These use real scoreboard barriers: the producing instruction allocates one, the first consumer of its result waits on it, and the warp is ineligible until the data lands. The stall-reason names in the counters map directly onto this split.

Long scoreboard — the expensive one

Long scoreboard is a warp waiting on a variable-latency memory dependency: it issued a global or local load and has reached the instruction that consumes the loaded value. This is the stall reason that dominates most memory-bound kernels, and the one worth chasing.

Its cost scales with where the data came from. A hit in L1 or L2 is tens of cycles; a trip to HBM is hundreds. Uncoalesced accesses make it worse twice over — more memory transactions and a longer wait before the last one returns.

The fixes are the familiar memory-hierarchy ones: improve coalescing, stage reuse through shared memory, widen loads to move more bytes per instruction, and — most usefully for the scheduler — hoist loads far above their consumers so the warp has independent work to issue while data is in flight. That last move is prefetching by another name, and it converts long-scoreboard stalls into useful issue slots without changing a byte of traffic.

Short scoreboard, barriers and throttles

Short scoreboard covers dependencies on the SM’s local shorter-latency variable pipes — shared memory, special-function units, and other traffic queued through the memory-input/output path. It behaves like a small long scoreboard: real waiting, but tens of cycles rather than hundreds. A kernel dominated by it is usually shared-memory bound, and the classic culprit is bank conflicts.

Barrier is a warp parked at a block-wide synchronisation point waiting for its siblings. High barrier stall means load imbalance inside the block or too many synchronisation points, and it is worsened by large blocks — the slowest warp sets the pace for all of them.

MIO throttle and the related instruction-queue throttles are different in kind: the warp’s data is fine, but the queue feeding a pipe is full. That is a bandwidth signal, not a latency one, and adding warps only makes the queue longer. You have to issue fewer instructions to that pipe.

Execution dependency — the ILP stall

Wait, sometimes described as an execution or fixed-latency dependency stall, is the warp sitting out the compiler-encoded gap between a math instruction and its immediate consumer. No memory is involved; the pipeline simply has not produced the result yet.

This is the purest instruction-level-parallelism signal there is. Code shaped as a chain — each operation consuming the previous one — can only issue one instruction every few cycles per warp, no matter how fast the math pipes are.

The cure is independent work in the same warp: unroll the loop, keep several partial accumulators instead of one, interleave two independent computations. Each of those gives the compiler instructions it can slot into the gap. Note that extra warps also cover this stall, because a different warp can issue during the gap — which is exactly why occupancy and ILP are substitutes rather than separate goals.

Not selected — the stall you want to see

Not selected means the warp was eligible and the scheduler chose someone else. Nothing is wrong. The instruction issued that cycle, just from a different warp.

A high not-selected share is therefore a report of surplus parallelism: you have more ready warps than issue slots, so the scheduler is rationing. Adding occupancy to a kernel in this state cannot help; the bottleneck has moved to instruction issue or to whichever pipe the selected warps are saturating. The productive responses are to do more per instruction — wider loads, tensor-core or vector operations instead of scalar ones — or to accept that you are issue-bound. It is also a case where reducing occupancy is often free: fewer warps with more registers each can raise ILP without emptying the eligible set.

The selection policy itself is not publicly documented; academic models describe it as greedy-then-oldest or a loose round-robin. The practical implication is that you control how many eligible warps exist, not which one wins.

Two ways to fill an issue slot — warps or ILP

The scheduler needs one eligible warp per cycle, and there are exactly two ways to supply it. Thread-level parallelism: more resident warps, so that while some wait, others are ready. Instruction-level parallelism: more independent instructions within each warp, so a single warp stays eligible across its own latencies.

They trade against each other through the register file, which is fixed per SM. Independent work needs live values, live values need registers, and more registers per thread means fewer resident warps. This is why the well-known result that some kernels run faster at lower occupancy is not a paradox: spending the register budget on unrolling and multiple accumulators can cover latency more cheaply than spending it on extra warps.

Roughly, latency to be hidden divided by the work in flight per warp gives the warps you need. Raise the numerator per warp and the required warp count falls.

Reading warp state, and what to change

Hardware samples warp state periodically, so profilers can report, per scheduler, the average cycles a warp spends in each stall reason per issued instruction. Read it in a fixed order rather than jumping at the biggest bar.

First check issue-slot utilisation and eligible warps per scheduler. If schedulers issue most cycles, the kernel is issue-bound and the histogram is mostly not-selected noise. If the eligible count is routinely below one, you have a real supply problem, and only then does the dominant stall reason tell you which lever to pull: long scoreboard means fix memory access or hoist loads; short scoreboard means fix shared-memory conflicts; barrier means rebalance or synchronise less; a throttle means reduce instructions to that pipe, never add warps; wait means add ILP.

Finally, attribute the stall to a source line before acting. Warp-state totals for a whole kernel usually hide the fact that one or two instructions own nearly all of the cost.