Why architecture matters here
GPUs are expensive. Idle SMs cost money; contended SMs cost latency. The architecture must balance those two forces. Streams let a single process pipeline compute and copy. Priorities steer streams to different queues. MPS lets multiple processes share a device without stepping on each other. MIG carves a card into hard slices with independent SMs and memory.
Without this scheduling plane, a single greedy client can consume an entire GPU, and every other tenant sees tail latency spikes and eviction. With it, you can offer bronze / silver / gold tiers, colocate training and inference safely, and predict tail behavior.
The plane is more than the primitives — it is the policies, the observability, and the placement logic that turn primitives into a service.
The architecture: every piece explained
The top strip captures the goal: share, isolate, and prioritize. Workload submitter is whatever launches kernels and memcpys — a training loop, an inference server, or a notebook. CUDA streams are asynchronous pipelines: a stream is an ordered queue of operations, and multiple streams execute concurrently if resources allow. Stream priority selects one of a few internal queues on the GPU; high-priority streams preempt low. Events + graphs are the record/replay primitive: capture a fixed sequence of kernels into a CUDA graph and launch it in a single call to remove CPU-side overhead.
The middle row is the multi-tenant strip. MPS server multiplexes streams from different processes onto one context so they can co-exist on the same SMs; SMs are time-sliced but the launch overhead vanishes. MIG partitions carve one A100/H100 into several smaller GPUs each with its own SMs, memory bandwidth, and cache — hard isolation, no noisy neighbor. SM allocation is a percentage quota per MPS client, letting a bronze tenant use at most 20% of SMs while a gold tenant gets 60%. Memory partition is the same idea applied to bytes of HBM.
The lower rows are the plane. Scheduler policy encodes the queueing discipline — FIFO by priority, fair share by tenant, preempt low-priority streams for latency-sensitive ones. Observability exports nsys traces, DCGM metrics, and per-kernel occupancy so you can prove the policy holds. Placement plane bin-packs jobs onto streams, MPS clients, or MIG slices given an SLO — training goes to MIG slices sized for its batch, inference goes to MPS with a stream-priority policy.
End-to-end flow
End-to-end: a scheduling request arrives for a 7B-parameter model inference with a 200 ms SLO. The placement plane consults free capacity: it sees a MIG slice on rack 3 with 30% headroom and assigns the workload. The container starts, opens a CUDA context on that MIG slice, and launches an inference server. Later a training job arrives asking for a whole H100 with 80 GB memory. The plane routes it to rack 5 outside the MIG partitioning; MPS is disabled there because training will consume the SMs. When a burst of high-priority inference requests arrives, streams on the MIG slices bump to high priority and finish first; low-priority background eval jobs on the same slice wait. Every step is traced in nsys; DCGM confirms SM allocation matches policy; the placement plane logs the decision for cost accounting.