Two completely different schedulers decide when your kernel runs, and conflating them is the source of most GPU scheduling disappointment. One lives on the die: a work distributor that places thread blocks on SMs, and per-SM warp schedulers that pick an eligible warp every clock. It has no policy knobs and no idea what a tenant is. The other is the software plane you actually control — streams, priorities, MPS, time-slicing, MIG, and the placement logic above them. The software plane cannot override the hardware; it can only shape the work the hardware is handed. This piece walks both levels, then gets concrete about sharing a device: what MPS really isolates, what a context switch costs, and how to choose between the three sharing mechanisms.
Two schedulers stacked on one device
The hardware scheduler operates on nanoseconds. It answers two questions continuously: which thread block should occupy a newly freed slot on an SM, and which resident warp should issue an instruction this cycle. It is greedy, it is not fair, and it exposes no interface. You influence it only indirectly — through grid size, through how many registers and how much shared memory a block asks for, and through which stream you launched into.
The software plane operates on microseconds to seconds: streams express ordering and permission to overlap, priorities express preference, and MPS, time-slicing and MIG decide how whole processes coexist.
The failure mode is expecting a software knob to produce a hardware behaviour the hardware never offered: stream priority does not partition SMs, MPS does not give you bandwidth QoS, time-slicing does not give you memory isolation. Knowing which layer owns a guarantee tells you whether you can have it at all.
The work distributor — blocks land on SMs
A kernel launch is a grid of thread blocks. The GPU's global work distributor (NVIDIA's GigaThread engine) hands blocks to SMs that have enough free registers, shared memory and warp slots to host them. The contract is deliberately weak: blocks are independent, they may run in any order on any SM, and a block cannot wait on another. Once resident, a block stays resident until it retires — then the distributor drops the next block into the vacated slot.
That greedy refill has a very visible consequence: wave quantisation. If a device holds, say, 160 blocks of your kernel at once, a 200-block grid runs as one full wave plus a nearly empty second wave, and costs roughly what a 320-block grid would. Sizing grids to whole waves, or using persistent kernels that launch one wave of long-lived blocks and pull work from a queue, is a scheduling optimisation the hardware will never do for you.
The warp scheduler — latency hiding at cycle granularity
Inside a resident block, threads are grouped into warps of 32 that execute in lockstep. A modern SM is divided into several processing partitions, each with its own warp scheduler. Every cycle a scheduler surveys its warps, finds one whose next instruction is eligible — operands ready, the required functional unit free, not parked at a barrier — and issues it.
The switch between warps is free. Each resident warp keeps its registers allocated for its entire lifetime, so there is no state to save or restore; the scheduler simply picks a different row. That is the GPU's entire latency-hiding strategy: where a CPU spends transistors on caches and speculation to avoid waiting on memory, a GPU just has other warps ready to issue while one waits hundreds of cycles for HBM.
Which means the scheduler needs candidates: a cycle in which every warp is stalled is an issue slot burned. How many warps can be resident is the occupancy question, covered separately.
Streams and priorities — the software handles
A stream is an ordered queue of operations; work in different streams is allowed to overlap. The semantics, events and CUDA graphs are covered in depth elsewhere on this site, so here only the scheduling consequence matters.
CUDA exposes stream priority as a small integer range you query with cudaDeviceGetStreamPriorityRange — a handful of levels, not a general-purpose nice value. Priority biases the work distributor: when blocks from several streams are queued and an SM slot frees up, the higher-priority stream's blocks are preferred. What it does not do is evict anything. Resident blocks from a low-priority kernel keep their slots until they retire, and no SMs are reserved.
So priority buys queueing preference measured in the duration of the blocks ahead of you. If your background kernel has 10 ms blocks, a high-priority launch can still wait 10 ms. Keep low-priority kernels short-blocked and priority starts to work.
Why concurrent kernel execution rarely materialises
The marketing picture shows several kernels running side by side on one device. In practice people open a profiler timeline and find them perfectly serialised. Four reasons dominate.
The first kernel already filled the machine. If a grid has enough blocks to occupy every SM there is no slot for anyone else, and the second kernel starts as the first drains. Concurrency is the reward for having too little work per kernel, not for optimisation.
Resources, not just slots. A second kernel can only start where a full block's worth of registers and shared memory is free, so a register-hungry kernel blocks co-residency even at low warp occupancy.
Accidental synchronisation. The legacy default stream synchronises with other blocking streams, so one stray synchronous copy or allocation serialises everything after it.
The framework submitted onto one stream anyway. Confirm overlap on a timeline rather than assuming the design produced it.
MPS — many processes, one context
Without MPS, two processes mean two CUDA contexts, and the GPU will not run kernels from two contexts simultaneously — the driver alternates between them. That is disastrous for a fleet of small inference processes that each use a fraction of a device.
The Multi-Process Service funnels work from many clients into a single shared context, so their kernels become genuinely co-resident on the SMs: they share the machine at block granularity instead of alternating in time. Two knobs matter — an active thread percentage per client, capping the fraction of the device its blocks may occupy, and a pinned device memory limit.
The isolation limits are real. The thread percentage is a ceiling, not a reservation. There is no memory-bandwidth or L2 partitioning, so a bandwidth-hungry neighbour still slows you down. On Volta and later each client gets its own GPU address space and far better fault containment than the old shared-address-space model, but clients still share hardware and a failure domain. MPS is a utilisation tool for workloads that trust each other, not a tenant boundary.
Time-slicing, context switches and preemption
Time-slicing is the default when several contexts want one GPU and MPS is not in play: the driver runs one context for a while, then swaps to the next. It needs no cooperation, which is why the Kubernetes device plugin's time-slicing mode — advertising one physical GPU as N schedulable replicas — is the easiest sharing scheme to turn on.
It is also the most expensive. A GPU context switch is nothing like a thread switch: the live state includes the register file and shared memory of every resident block, so the driver either drains blocks or spills a large amount of state. Older architectures could only switch at block boundaries, so one long kernel could hold the device; Pascal added instruction-level compute preemption, letting the hardware stop warps mid-stream and resume them later. That is a robustness fix, not a QoS knob: it acts between contexts at the driver's discretion, and stream priority does not trigger it.
Two consequences bite. Latency-sensitive inference becomes unpredictable, because a request can arrive at the start of somebody else's slice and preemption is too heavyweight to rescue it. And time-slicing partitions time only, never memory: every context holds its allocations simultaneously, so N replicas sharing one card must still fit in one card's HBM.
Choosing between MPS, time-slicing and MIG
Three mechanisms, three very different isolation stories. MIG carves a supported datacentre GPU into instances with their own SMs, cache slices and memory paths; the partitioning details are a topic of their own, so this is only about when to reach for it.
| Mechanism | What it shares | Isolation | Reach for it when |
|---|---|---|---|
| Streams + priorities | One process, one context | None — cooperative | You control all the work and want overlap |
| MPS | Many processes, one context, co-resident blocks | Soft: SM% ceiling, shared bandwidth | Several trusted processes each underfill the GPU |
| Time-slicing | Many contexts, alternating in time | Temporal only, no memory split | Latency does not matter; dev and batch work |
| MIG | Nothing — hardware partitions | Hard, with predictable performance | Untrusted tenants or a real SLO to sell |
A workable default: streams inside your own process first, because they are free; MPS when several cooperating processes each leave the device idle; MIG when a tenant boundary or an SLO is involved and you can live with fixed instance sizes and a reconfiguration step; time-slicing only where predictability does not matter.
From primitives to a scheduling plane
Primitives are not a policy. A scheduling plane classifies each workload into a tier, chooses the mechanism that fits, places the job, and then proves the policy held. Placement bin-packs jobs given their SLO: latency-critical serving goes to a hard partition, best-effort evaluation goes wherever there is slack.
Verification is the part teams skip. DCGM gives per-process utilisation, SM activity and memory, which is how you catch an MPS client exceeding its share. A profiler timeline confirms the overlap you designed actually happens, and per-kernel duration histograms expose the long-blocked background kernel defeating your stream priorities.