GPU Scheduling With Slurm — HPC Job Scheduler

Two schedulers, one job — where Slurm stops and the driver begins

Two schedulers sit between a user's command and a kernel executing on an SM, and they share no vocabulary. The cluster scheduler — Slurm — works in units of jobs, nodes, walltime and countable resources. Its decision is which node, which physical devices, starting when. Everything below that boundary, once the process is running, belongs to the driver and the GPU itself: which thread block lands on which SM, which warp issues this cycle, whether two processes can be co-resident. Slurm has no visibility into any of it.

The consequence is that Slurm's isolation is allocation-time, not runtime. It can guarantee that no other job is handed device 3 on node gpu07. It cannot guarantee that the job holding device 3 gets a fair share of HBM bandwidth, or that a co-scheduled job on devices 0–2 will not saturate the PCIe root complex you share. If you need performance isolation inside a card rather than between cards, that is a partitioning question — see MIG and GPU sharing strategies — and Slurm's role is only to expose the resulting partitions as schedulable units.

Three daemons implement the cluster side. slurmctld on the control node holds the queue and makes every placement decision. slurmd on each compute node reports what hardware it found, launches job steps, and enforces the resource cage. slurmdbd stores accounting and the association tree that fairshare and QoS limits are computed from. A GPU cluster that misbehaves is almost always misbehaving in exactly one of those three places, and knowing which one narrows the search enormously.

GRES — turning a card into something the controller can count

Slurm's Generic RESource (GRES) subsystem is how anything that is not a core, a socket or a byte of RAM becomes schedulable. GPUs are the canonical case. It takes two files, and they must agree with each other and with reality.

# slurm.conf -- the controller's view of the cluster
GresTypes=gpu
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
NodeName=gpu[01-16] CPUs=128 Sockets=2 RealMemory=1000000 Gres=gpu:a100:8

# gres.conf -- the node's view of its own devices
AutoDetect=nvml
# or, spelled out explicitly:
# Name=gpu Type=a100 File=/dev/nvidia[0-3] Cores=0-63
# Name=gpu Type=a100 File=/dev/nvidia[4-7] Cores=64-127

SelectType=select/cons_tres is not optional detail. It is the consumable-trackable-resources selector, and it is what lets a node be shared between jobs at the granularity of individual GPUs and lets per-task GPU flags mean anything. On a cluster still running the older core-only selector, half the request syntax below silently degrades.

The Type string (a100, h100, l40s) is user-visible and requestable, which is what makes a heterogeneous fleet usable. AutoDetect=nvml asks the NVIDIA Management Library to enumerate devices and their CPU affinity rather than trusting a hand-written file — strongly preferred, because the failure mode of the manual form is a silent mismatch. If slurmd finds fewer devices than slurm.conf promised — a card fell off the bus, a driver upgrade half-applied — the controller drains the node rather than schedule jobs onto phantom GPUs. That drain is a feature; the alarming version is a node that stays in service because the config was written loosely enough to match a degraded machine.

Requesting GPUs — what each flag actually binds

There are two generations of GPU request syntax and they coexist. The older --gres=gpu:2 form means two GPUs per node. The newer family — --gpus, --gpus-per-node, --gpus-per-task, --gpus-per-socket — is explicit about the denominator, and --gpus=16 means sixteen for the job in total, however they are distributed. Mixing the two in one script is the fastest way to get an allocation you did not intend.

#!/bin/bash
#SBATCH --job-name=llama-sft
#SBATCH --partition=gpu
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-task=1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-gpu=100G
#SBATCH --time=08:00:00
#SBATCH --gpu-bind=closest
#SBATCH --output=logs/%x-%j.out

srun python train.py --config configs/sft.yaml

--mem-per-gpu and --cpus-per-gpu exist because the sizing that matters on a GPU node is per-device, not per-node: a job asking for two of eight GPUs should get roughly a quarter of the host RAM and a quarter of the cores, and expressing that as a fixed --mem breaks the moment someone runs the same script on a different SKU. --ntasks-per-gpu covers the inverse case, several ranks sharing one device.

Two habits pay for themselves. First, request the Type (--gres=gpu:a100:2) only when the code genuinely needs it — an unnecessary type constraint on a mixed fleet turns a five-minute wait into an overnight one. Second, ask --exclusive for anything doing multi-node collectives, because a co-tenant on the same node competes for the NIC and the PCIe path your all-reduce depends on, and Slurm's accounting will not show you that as contention.

Enforcement — cgroups, ConstrainDevices, and CUDA_VISIBLE_DEVICES

An allocation is a promise. The cage that makes it true is Linux cgroups, and it is configured separately from everything above.

# slurm.conf
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity
PrologFlags=Contain

# cgroup.conf
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainDevices=yes

ConstrainDevices=yes is the single most consequential line in a GPU cluster's configuration, and it defaults to off. With it, the job's cgroup permits access only to the /dev/nvidia* nodes it was allocated; anything else fails at open(). Without it, Slurm still sets CUDA_VISIBLE_DEVICES, but that variable is advisory — it is read by the CUDA runtime and can be overwritten by any user process, any launcher script, or any framework that helpfully sets it for you. A single job that clears it can then open all eight devices on a node it was allocated one of, and the corrupted neighbours will report the failure as a mysterious out-of-memory error in someone else's training run.

The second surprise follows from the same mechanism: with device constraint active, CUDA_VISIBLE_DEVICES holds allocation-local indices. A job given physical devices 4 and 5 sees 0,1. This is correct and desirable — code should never hardcode a global index — but it is why every job on the cluster appears to be using "GPU 0", and why correlating a job against nvidia-smi output on the node requires SLURM_JOB_GPUS (the physical list for the allocation) rather than the CUDA variable. SLURM_GPUS_ON_NODE gives the count, which is the portable way for a launcher to size its process group.

Affinity — putting the GPU next to the cores that drive it

Getting the right number of GPUs is the easy half. Getting GPUs that are close to the cores and the NIC the job will use is where real throughput is won or lost. On a two-socket node, the PCIe devices hang off specific root complexes; a rank pinned to socket 0 driving a GPU on socket 1 pays a cross-socket hop on every host-to-device transfer, and the penalty shows up as an unexplained input-pipeline stall rather than as anything labelled "affinity".

Slurm learns the topology from the Cores= field in gres.conf — the local CPU cores that share a socket with each device — or derives it automatically under AutoDetect=nvml. That field is a hint used at allocation time: when a job asks for four GPUs and sixty-four cores, the selector prefers a set whose cores and devices are on the same socket. A hand-written Cores= mask that is wrong is worse than none at all, because the scheduler will confidently make bad placements and nothing will report an error. This is a strong argument for AutoDetect.

At launch time, --gpu-bind decides how the allocated devices are handed to individual tasks. --gpu-bind=closest gives each task the device nearest its CPU binding, which is what most multi-rank training wants. single:1 restricts each task to exactly one device. map_gpu: and mask_gpu: give explicit per-task assignments for the cases where a framework has its own opinion about ordering. Adding verbose makes srun print what it actually bound, which is the fastest way to settle an argument about whether the topology flags took effect.

Partitions and QoS — where the policy actually lives

Partitions are Slurm's queues, and on a GPU cluster they carry most of the policy. A partition definition sets who may submit, the walltime ceiling, the default and maximum node counts, and — importantly — a PriorityTier that governs which partition's jobs may preempt which.

PartitionName=gpu Nodes=gpu[01-16] MaxTime=24:00:00 PriorityTier=10 \
    TRESBillingWeights="CPU=1.0,Mem=0.1G,GRES/gpu=32.0" QOS=normal
PartitionName=preempt Nodes=gpu[01-16] MaxTime=04:00:00 PriorityTier=1 \
    PreemptMode=REQUEUE

Two partitions over the same nodes is the standard pattern for mixing guaranteed and opportunistic work: a high-tier partition for funded projects and a low-tier scavenger partition whose jobs run on whatever is idle and are requeued the moment the high tier needs the hardware.

QoS objects, managed with sacctmgr, layer orthogonal limits on top and are where per-GPU caps live. GrpTRES=gres/gpu=64 caps the whole QoS at sixty-four concurrent GPUs; MaxTRESPerUser=gres/gpu=8 stops one user filling the cluster; MaxWall, MaxJobsPerUser and a Priority boost complete the toolkit. A UsageFactor below one makes a QoS cheap in the fairshare ledger — the usual way to price a preemptible tier so that using it does not consume a group's allocation at full rate. Set Flags=DenyOnLimit if you would rather a violating job be rejected at submission than sit pending forever with a limit-related reason code that nobody reads.

Fairshare — and the billing weight that makes GPUs cost what they cost

When more jobs are runnable than there are GPUs, order is decided by the multifactor priority plugin: a weighted sum of age, fairshare, QoS, partition, job size and TRES-specific terms, recomputed continuously. sprio -l shows the breakdown for a pending job, and it is the only honest answer to "why is my job behind theirs".

Fairshare compares each account's recent usage against its share of the cluster, with usage decaying on a half-life set by PriorityDecayHalfLife. The critical detail on a GPU cluster is what "usage" counts. Billing is computed from TRESBillingWeights on the partition, and if that line omits GRES/gpu, a job holding eight A100s for a day bills the same as a job holding the node's cores and no GPUs at all. Fairshare then rewards exactly the wrong behaviour, and no amount of tuning the weights above will fix it. A common convention sets the GPU weight so that one GPU bills like the fraction of the node it represents — on a 128-core, 8-GPU node, GRES/gpu=16 makes a GPU worth sixteen cores.

The same omission has a second, quieter form. AccountingStorageTRES in slurm.conf must list gres/gpu (and any per-type entries such as gres/gpu:a100) or slurmdbd records no GPU usage whatsoever. sacct -o JobID,AllocTRES,Elapsed then reports jobs that used forty GPU-hours as having used none, utilisation dashboards read zero, and the capacity case for buying more hardware evaporates. Both settings are easy to add on day one and painful to backfill, because history cannot be recomputed.

Backfill — the scheduler that fills the holes, and why your job misses it

Slurm's main scheduling loop walks the queue in priority order and starts what fits. On its own that wastes enormous amounts of GPU time: a 32-node job at the head of the queue reserves a future start time, and the cluster drains toward it while small jobs that would fit in the gap wait behind. The backfill scheduler exists to fill those holes — it starts lower-priority jobs provided they finish before the reserved start time of the job they jumped, so no job is delayed by the favour.

SchedulerType=sched/backfill
SchedulerParameters=bf_window=2880,bf_resolution=300,bf_max_job_test=2000,bf_continue

The mechanism has one hard prerequisite: accurate walltimes. Backfill can only prove a job fits in a hole if it knows how long the job runs, and the only number it has is --time. A user who omits the flag inherits the partition default — often the maximum — and becomes permanently unbackfillable, waiting for a full-size opening that a twenty-minute job never needed. Teaching users to request a realistic walltime is the highest-leverage scheduling intervention available on most GPU clusters, and it costs nothing.

The tuning knobs trade throughput for controller CPU. bf_window is how far ahead the planner looks (it should exceed the longest walltime the partition allows, or long jobs are invisible to it), bf_resolution is the time granularity, and bf_max_job_test caps how many pending jobs are considered per pass. On a deep queue, a too-small bf_max_job_test means the backfill pass never reaches most of the queue; too large and slurmctld spends its cycles planning instead of dispatching. sdiag reports the backfill loop's timing and depth, which is how you find out which of the two you have.

Preemption and requeue — reclaiming GPUs without losing the work

Preemption is how a cluster sells the same GPUs twice: guaranteed capacity to one tier, opportunistic capacity to another. PreemptType=preempt/partition_prio lets a higher PriorityTier partition displace a lower one; PreemptType=preempt/qos does the same by QoS. PreemptExemptTime guarantees a minimum run before a job is eligible to be preempted, which stops a scavenger job from being killed thirty seconds in, forever.

PreemptMode decides what displacement means, and the GPU-specific trap is here. SUSPEND stops the job's processes but leaves them resident — which for a GPU job means the CUDA context and every byte of device memory stay allocated. The preempting job arrives to find the HBM already spoken for and fails to allocate. Suspension is a workable mode for CPU jobs and a poor one for GPU jobs; REQUEUE, which kills the job and returns it to the queue, is the usual GPU choice. CANCEL is the harsher variant that does not resubmit. GANG time-slices between jobs and inherits the same device-memory problem.

Requeue only helps if the job can resume, which pushes the burden onto the workload: checkpoint on a cadence, write to shared storage, and detect a restart on startup. Slurm gives the job a warning signal --signal=B:USR1@120 some seconds before the kill, and a GraceTime on the partition sets how long it waits — enough to flush a checkpoint if, and only if, the checkpoint is small enough to write in that window. The economics of that trade, and what a training checkpoint actually contains, are worked through in GPU preemption handling. Also note #SBATCH --requeue is not the default everywhere: a job that is not marked requeueable simply dies.

Topology-aware allocation for multi-node training

For a single-node job, placement ends at the device. For distributed training, which nodes matter as much as how many, because an all-reduce runs at the speed of its worst link. Sixteen nodes spread across four leaf switches with an oversubscribed spine will finish an epoch dramatically slower than sixteen nodes under one leaf, and Slurm will happily give you either unless it is told the fabric exists.

# topology.conf
SwitchName=leaf1 Nodes=gpu[01-08]
SwitchName=leaf2 Nodes=gpu[09-16]
SwitchName=spine Switches=leaf[1-2]

# slurm.conf
TopologyPlugin=topology/tree

With the tree plugin the selector prefers node sets that minimise switch count. Users can make the preference explicit with --switches=1@30:00: allocate within a single switch, but give up and take a worse placement after thirty minutes of waiting rather than pend indefinitely. That second argument is what makes the flag safe to recommend — a bare --switches=1 on a busy cluster is a request to wait forever.

Inside the node, the rail structure of the NVLink and InfiniBand fabric decides whether an allocation is fast or merely correct, and that is where --gpu-bind=closest and NCCL's own topology detection meet. Slurm hands over a set of devices and a CPU mask; NCCL then discovers the interconnect and builds rings or trees over it. The failure signature of a bad hand-off is a job that runs, produces correct results, and gets a fraction of the expected bus bandwidth. Confirming what the fabric actually offers is a separate exercise — see NCCL collectives and InfiniBand and NVLink.

Containers, prologs and the node that quietly went bad

HPC clusters have no Docker daemon and no intention of acquiring one, so the container story on Slurm is rootless and image-as-file. Enroot converts an OCI image into an unprivileged, flattened root filesystem; the pyxis SPANK plugin wires it into srun so a container becomes a launch flag rather than a wrapper script.

srun --container-image=nvcr.io#nvidia/pytorch:24.01-py3 \
     --container-mounts=/scratch/$USER:/workspace \
     --container-workdir=/workspace \
     python train.py

The advantage over a wrapper is that the container is created per task, inside the job's cgroup, with the GPU allocation and the environment already applied — so CUDA_VISIBLE_DEVICES, the device cgroup and the CPU binding all still hold. The driver's user-space libraries come from the host, not the image, which is the same constraint every GPU container faces and the reason an image built against a newer CUDA runtime than the cluster's driver supports fails at startup; the compatibility rules are worked through in GPU-enabled containers.

The operational counterpart is node health. HealthCheckProgram runs a script on every node at HealthCheckInterval, and on a GPU cluster it should do more than ping: query each device for ECC errors, confirm the expected device count, check that persistence mode is set and that no orphaned process holds memory. A node that fails should be drained, not left to accept the next job and fail it. Symmetrically, an Epilog that verifies GPUs return to idle catches the leaked process — the one that survives the job and holds several gigabytes of HBM — before it silently poisons every subsequent allocation on that node.

Diagnosing the five complaints you will actually get

Most GPU-cluster complaints reduce to a handful of shapes, and each has a command that identifies it in one step.

"My job never starts." squeue -o "%.18i %.9P %.8j %.2t %.10M %R" prints the reason. Resources means the hardware is genuinely busy; Priority means something outranks you and sprio -l will say what; a QoS- or association-limit reason means you hit a cap, not a shortage, and adding nodes will not help. scontrol show job JOBID gives the full request, which frequently reveals a typed GRES constraint the user forgot they wrote.

"The cluster looks empty but nothing runs." sinfo -o "%20N %10T %10G %.6D" shows node states and GRES. Drained nodes with a GPU-related reason are the usual culprit; sinfo -R lists the reasons in one place. A node in drain after a driver update, with configured GRES that no longer matches what slurmd detects, will hold out an entire rack of capacity.

"My code sees the wrong GPUs." Compare SLURM_JOB_GPUS (physical) with CUDA_VISIBLE_DEVICES (allocation-local). If the latter is missing or shows more devices than requested, check that ConstrainDevices=yes is actually set on that node — a per-node cgroup.conf drift is common.

"We cannot show utilisation." Verify AccountingStorageTRES includes gres/gpu, then use sacct -a -o JobID,User,AllocTRES,Elapsed,State for allocated GPU-hours. Note that this measures allocated, not used: a job holding eight GPUs at four percent occupancy looks identical to a saturating one. Pairing Slurm accounting with per-device telemetry from DCGM is the only way to see the difference, and on most clusters the gap between the two numbers is the largest single source of wasted GPU capacity.