GPU-Enabled Containers — Running GPU Workloads in Docker

Why a GPU container is not a normal container

A normal container is a promise that the image is the environment. Copy the image to any host with a compatible kernel and the process sees the same libraries it saw on your laptop. GPU containers break that promise on purpose, because the piece that talks to the hardware is not relocatable.

The NVIDIA stack is really three layers. At the bottom is the kernel module set — nvidia.ko, nvidia-uvm.ko, nvidia-modeset.ko — loaded on the host and owning the PCIe device. Above it sits a userspace driver library, libcuda.so, which speaks a private ioctl protocol to that module. The two are released together and are versioned together; a userspace driver library from one release will not negotiate with a kernel module from another. Above that sits the CUDA runtime, libcudart.so, plus the math libraries (cuBLAS, cuDNN, NCCL) and your framework. Only the top layer is safely portable.

The dividing line, then, is not “driver versus application”. It is libcuda.so. Everything below it must come from the host because it is glued to the running kernel module; everything above it can and should be baked into the image. An image that bundles its own libcuda.so is not more portable — it is a time bomb that will load against the wrong kernel module on the first host whose driver differs. Base images from the CUDA family are deliberately built without it, which is why an ldd inside a freshly built GPU image shows libcuda.so.1 as not found and the same image works fine once the runtime hook has run.

What the container runtime hook actually injects

Docker’s --gpus flag and the Kubernetes device plugin both end up in the same place: a hook that runs after the container’s filesystem is prepared but before its entrypoint is executed, with the container’s mount namespace still writable. The hook queries the host driver, works out the exact set of files that belong to it, and mounts them in.

Three categories of thing get injected. First, device nodes: the character devices under /dev that represent the control interface and each individual card, plus the unified-memory devices. Without them the container has no file descriptor to open and every CUDA call fails at initialisation, no matter how complete the libraries are. Second, driver userspace: the versioned libcuda.so, the management library behind nvidia-smi, the NVML and encode/decode libraries, and the nvidia-smi binary itself, followed by an ldconfig run so the dynamic linker inside the container finds them. Third, capability gating: an environment contract that decides how much of the driver surface to expose, so a pure-compute container does not receive the graphics and display libraries it will never call.

The contract is expressed with environment variables that the image itself usually declares. NVIDIA_VISIBLE_DEVICES chooses which physical devices the hook exposes — all, a count, a list of indices, or, best of all, a list of GPU UUIDs. NVIDIA_DRIVER_CAPABILITIES chooses which library groups to inject; compute,utility is the pairing that a training or inference container actually needs. Because these are ordinary environment variables baked into base images, a container that inherits NVIDIA_VISIBLE_DEVICES=all and is started under a GPU-aware runtime will quietly grab every card on the node, which is a common way for a sidecar or a debug shell to end up holding hardware nobody meant to give it.

Newer deployments describe the same injection declaratively through the Container Device Interface: a JSON specification on the host that names the device nodes, the bind mounts and the environment for each addressable device, which any conforming runtime can apply. It moves the logic out of a vendor-specific hook and into data, which matters most on nodes that mix accelerator vendors.

Driver version versus CUDA toolkit version

Four numbers are in play and people routinely conflate the first two. There is the kernel module version, the userspace driver version (these two are one release and must be identical), the CUDA toolkit version compiled into the image, and the compute capability of the physical silicon. Only the toolkit number is yours to choose in the Dockerfile.

The default relationship is backward compatibility in one direction: a driver supports the toolkit it shipped with and every older toolkit. Ship an image built against an older toolkit onto a node with a newer driver and it runs. Ship an image built against a newer toolkit onto a node with an older driver and the CUDA runtime refuses at initialisation, because the runtime asks the driver for entry points that release does not export. The error text names insufficient driver support rather than a missing symbol, which is why people go looking for the wrong problem.

The forward-compatibility escape hatch

There is a supported way to run a newer toolkit on an older driver, and it is worth understanding precisely because it is narrow. NVIDIA publishes a compatibility package containing an alternative userspace driver library, new enough for the new toolkit, that still speaks the older kernel module’s protocol. Placed on the loader path ahead of the injected host library, it lets a newer CUDA runtime initialise against an older kernel module. Two constraints bite. It is supported only on the datacenter driver branches, not on the consumer or workstation branches, so it is not a way to make a laptop match a cluster. And it is a compatibility shim, not a time machine: it cannot conjure hardware features that the installed kernel module has no code path for.

Compute capability is the fourth number and behaves differently again. It is a property of the silicon, and it decides whether a compiled kernel can execute at all. A binary built without a code path for the deployment GPU either falls back to a just-in-time compile from embedded intermediate code or fails outright with no kernel image available. That is a build-flag question rather than a container question, and GPU CUDA Programming works through the compilation path in detail.

Why the works-on-my-machine argument fails harder here

Containers are supposed to end that argument, and for CPU workloads they mostly do. Here the image pins only the half of the stack you can see. The other half is a per-node property that no manifest records, that a fleet upgrades gradually, and that differs between your workstation, the CI runner and the production pool. A team can reproduce a bug on every machine they own and still not reproduce it on the node that pages them.

The failure also lands late. The container starts, the image layers unpack, Python imports, and only at the first CUDA context creation — sometimes minutes in, after a dataset has been read — does the mismatch surface. Health checks that probe an HTTP port pass happily in the meantime. The cheap fix is a startup probe that forces a real device operation: allocate a small tensor on the GPU and read a value back, then fail fast and loudly if it throws. Printing the driver version, the toolkit version and the device name into the first log line of every container costs nothing and removes most of the guesswork from the eventual incident.

Image size is a real operational problem

GPU images are enormous by ordinary standards. A framework image with the CUDA runtime, cuDNN, NCCL and a deep-learning framework is measured in gigabytes before your code is added; add a full toolkit with the compilers and headers and it grows again. That size is not merely untidy, it is latency in the one place you cannot afford it: a node that has just been autoscaled into existence must pull the image before the first pod runs, and on a cold node the pull frequently dominates every other part of startup. Scaling out to absorb a traffic spike is pointless if capacity arrives ten minutes late.

Three concrete moves cut it down. Pick the right base variant. The CUDA image family is published in tiers — a bare base, a runtime tier with the math libraries, and a development tier with the compilers and headers. Only a build needs the development tier; shipping it to production doubles the image so that a machine which will never compile anything can carry nvcc. Use a multi-stage build. Compile custom kernels or install build-time dependencies in the development stage, then copy only the resulting artifacts into a runtime-tier final stage.

Do not ship the CUDA libraries twice. This is the modern version of the problem and it is easy to miss: framework wheels installed from the Python index now pull their own packaged copies of the math and communication libraries as transitive dependencies. Install such a wheel into an image that already carries those libraries from the base layer and both copies land in the image, one of them dead weight. Decide which source of truth you want — base image or wheels — and build against it consistently.

Layer order is the other half. Docker caches per layer and invalidates everything below the first change, so the ordering that keeps rebuilds fast is: base, then system packages, then the dependency manifest and its install, then model weights if they are baked in, then application source last. Editing a Python file should invalidate one small layer, not trigger a reinstall of the framework. The same ordering pays at pull time, because unchanged layers are already on any node that ran a previous version.

# stage 1: build - needs the compilers, ships nothing
FROM nvidia/cuda:TAG-devel AS build
COPY requirements.txt .
RUN pip wheel --wheel-dir /wheels -r requirements.txt
COPY kernels/ /src/kernels/
RUN cd /src/kernels && python setup.py bdist_wheel -d /wheels

# stage 2: runtime - no nvcc, no headers, no build cache
FROM nvidia/cuda:TAG-runtime
COPY --from=build /wheels /wheels
RUN pip install --no-index --find-links=/wheels /wheels/*.whl && rm -rf /wheels
# application source last so an edit invalidates only this layer
COPY src/ /app/
ENV NVIDIA_VISIBLE_DEVICES=void NVIDIA_DRIVER_CAPABILITIES=compute,utility
ENTRYPOINT ["python", "/app/serve.py"]

Defaulting NVIDIA_VISIBLE_DEVICES to nothing in the image is a deliberate safety choice: the orchestrator sets it explicitly for the containers that are meant to hold hardware, and a container started by accident under a GPU-aware runtime gets no devices instead of all of them.

Device selection and what the container can see

Two variables with confusingly similar names operate at different layers, and knowing which is which resolves most selection bugs. NVIDIA_VISIBLE_DEVICES is read by the runtime hook on the host and decides which device nodes exist inside the container at all. CUDA_VISIBLE_DEVICES is read by the CUDA runtime inside the container and filters, within that already-restricted set, which devices the process enumerates. The first is a boundary; the second is a preference. Setting the second to a card the first never injected simply yields a process that sees no devices.

Renumbering is the trap that follows. The container always sees its devices numbered from zero. A pod granted the host’s fourth card sees a single device at index 0, so logs, metrics and error messages that report “GPU 0” are container-local and cannot be joined against host-level indices without a translation. Anything that must survive that boundary should key on the device UUID, which is stable, rather than on an index, which is not — host enumeration order itself is a function of PCI bus ordering and can change across a reboot or a firmware update.

Visibility is also asymmetric in a way that surprises people. nvidia-smi run inside a container reports the whole physical device: its total memory, its utilisation, its temperature. Those numbers include work from every other tenant on the card, and the process list is frequently blank or unresolvable because the PIDs belong to another namespace. Two practical consequences follow. Frameworks that size their memory pool as a fraction of “total device memory” will size against the whole card even when they are meant to share it, so the fraction has to be set explicitly per tenant. And any dashboard that attributes utilisation to a workload by reading the device from inside one container is measuring its neighbours as well as itself.

Kubernetes device plugins and the integer-resource wall

Kubernetes has no built-in notion of a GPU. It has a plugin interface: a daemon on every node opens a socket in a well-known directory, registers itself with the kubelet, and then serves two calls. A long-lived streaming call publishes the list of device IDs on that node and their health, which is how the node comes to advertise a vendor-named resource in its allocatable set. An allocation call is invoked when the kubelet is about to start a container that was granted devices; the plugin returns the device nodes to expose, the mounts to make and the environment to set, and the kubelet passes that down to the container runtime — which is precisely how the injection described earlier gets triggered on a cluster rather than by a docker run flag.

The consequence people trip over is the resource model. A GPU is exposed as an extended resource, and extended resources are integers. They cannot be fractional the way CPU can be expressed in millicores, they cannot be overcommitted, and the request must equal the limit — there is no burstable class for hardware. The smallest thing a pod can ask for is therefore one whole advertised unit. Every sharing scheme in the ecosystem exists to change what one unit means, because the scheduler’s arithmetic itself is not negotiable.

On a real cluster the plugin is rarely installed alone. An operator bundles the pieces that must agree with one another — a driver container that builds and loads the kernel module against the running node kernel, the container toolkit that installs the runtime hook and rewrites the runtime configuration, the device plugin, a feature-discovery daemon and a metrics exporter — and reconciles them as a unit. The alternative is baking the driver into the node image, which is simpler to reason about and much slower to change. The tradeoff is real: an operator makes the driver a cluster-managed artifact you can roll, while a baked image makes it a node-image artifact you must re-provision.

Sharing a card, and the isolation each option gives up

Three mechanisms let more than one container use one physical card, and from the container’s point of view they differ in what the pod requests and what the process sees. Under time slicing the plugin simply advertises more units than there are cards; each container opens its own context and the hardware interleaves them, with no memory isolation, no share guarantee and one shared fault domain. Under MPS the containers funnel their work through a shared server process so kernels from different tenants can genuinely run concurrently, which raises utilisation but keeps them inside one failure boundary. Under MIG the card is partitioned in hardware and each container is handed what looks like a smaller, entirely separate GPU with its own memory and its own slice of the memory path.

The choice is an isolation question, not a density question, and it deserves more than a paragraph: GPU Sharing Strategies scores the options against each other, GPU Time Slicing covers replica ratios and the failure modes they invite, NVIDIA MIG covers hardware partitioning, and NVIDIA vGPU covers the virtual-machine path. The container-side rule of thumb is short: if a workload has a latency objective or a memory floor you would defend in a review, give it a boundary the hardware enforces; if it is a notebook, a CI job or a development sandbox, density is worth more than isolation.

Labelling and scheduling a heterogeneous fleet

Fleets do not stay uniform. Cards get added a generation at a time, memory capacities differ within a generation, and a cluster ends up holding several device types whose only common property is that they are all called a GPU by the scheduler. An integer resource cannot express “this job needs eighty gigabytes of device memory”, so the difference has to be carried in labels.

A feature-discovery daemon writes the useful ones onto each node: the product name, the device memory, the compute capability, the count, the driver branch, and whether the card is partitioned. Workloads then select on those labels rather than on instance types, which keeps manifests portable across clouds and across hardware refreshes. Two habits make this work in practice. Select on the property, not the product — a job that genuinely needs a compute capability floor should say so, because a product-name selector silently excludes the next generation you buy. And taint every GPU node so that only pods carrying the matching toleration land there, otherwise ordinary CPU workloads pack themselves onto the most expensive machines you own and block the pods that actually need them. The node-pool and autoscaler mechanics underneath are cluster-level concerns covered in GKE architecture.

One consequence loops back to the image: if the fleet spans compute capabilities, the binaries in the image must too. A container that lands on a generation its kernels were never compiled for will either pay a just-in-time compile on first load or fail outright, and the node label that would have prevented it is only useful if something is actually selecting on it.

Health checking and the degraded device

A GPU can be present, enumerable, advertised as allocatable, and still be unable to finish a job. This is the failure class that makes GPU nodes different from CPU nodes: partial hardware failure is common, and it is usually silent until a workload hits it.

The signals worth watching are hardware-level, not application-level. The driver logs device errors to the kernel ring buffer with a numeric class, and the class distinguishes a benign application fault from a card that has genuinely misbehaved — the ones that indicate a hardware fault or a fallen-off-the-bus device are node-level events, not pod-level ones. Correctable memory errors accumulating quickly on one card are a leading indicator; an uncorrectable one is an immediate stop, because the affected memory row must be retired, and retirement only takes effect after a device reset. Falling clocks paired with a thermal or power throttle reason mean the card is alive but no longer delivering the throughput your capacity model assumed.

The plugin’s health stream is the mechanism that closes the loop: a device reported unhealthy is withdrawn from the node’s allocatable set, so the scheduler stops placing new pods on it. That handles new work but not existing work, which is why the operational pattern is a validation job that runs before a node joins the pool and a detector that runs continuously afterwards, taints the node on a hardware event, and lets a drain move the pods off. Validate before scheduling rather than after, and treat the diagnostic suite as a gate for a returning node rather than a debugging tool used once something has already failed.

Topology awareness for multi-GPU jobs

A pod that asks for eight units on a node with eight cards gets all of them and topology is moot. A pod that asks for two on a node with eight is where placement starts to matter, because the pairs are not equivalent. Two devices on the same high-speed fabric domain exchange gradients at fabric bandwidth; two devices that must route through the host bridge exchange them at a small fraction of that, and a collective-heavy job can lose a large share of its throughput to a placement decision that nothing in the pod spec mentioned. Plugins can prefer topologically close sets when allocating, and it is worth confirming that behaviour is enabled rather than assuming it.

Alignment does not stop at the accelerators. The host memory a device streams from, the CPU cores running the data loaders and the network interface carrying collectives all sit on a NUMA node, and a container whose CPUs are on one socket while its device hangs off the other pays for every transfer twice. The kubelet’s topology manager exists to align device, CPU and memory allocations onto one NUMA node, and enabling it is usually a bigger win for data-loader-bound training than any framework tuning. Two container-specific details finish the picture: shared-memory-based data loaders need the container’s /dev/shm raised well above the small runtime default or workers die with opaque bus errors, and multi-node collectives need the interconnect devices exposed into the container too, not just the GPUs. The fabric properties themselves are covered in NVLink and NVSwitch and PCIe host interconnect.

Operational failure modes

A driver upgrade takes the whole node with it

Because the userspace driver is injected from the host at container start, every running container on a node is pinned to the driver that was loaded when it started. Upgrading means unloading the kernel module, and the module will not unload while any process holds a handle to the device — which every GPU container does, for its entire life. So the upgrade is a drain, not a rolling restart: cordon the node, evict every GPU pod, unload, install, reload, uncordon. Containers that survive the swap in place would be running against a userspace library that no longer matches the module, which is worse than being restarted. Treat the driver version as a node-pool property and roll pools, and remember that long training jobs must checkpoint frequently enough to make a drain cheap.

A leaked process holding memory

Killing a container does not always free the device. If a process escapes the container’s cgroup, or a child survives its parent, or the runtime kills the entrypoint while a CUDA context is still being torn down, the device keeps memory allocated to a context with no visible owner. The symptom is a card reporting gigabytes in use with an empty process list, and the next pod scheduled onto it fails to allocate. From inside a container you usually cannot even identify the culprit, because the PIDs belong to another namespace — diagnosis has to happen on the host. The mitigations are unglamorous: make the entrypoint the container’s init process so signals reach it, give termination enough grace period to release contexts cleanly, and have the node-level detector treat “memory allocated, no owning process” as a condition worth acting on.

Errors that require a reset, and limits that count the wrong memory

Some faults are only clearable by resetting the device, and a reset requires exclusive access — so it fails while any container holds the card, which means the node must be drained first. Building that path deliberately (detect, taint, drain, reset, validate, return) is far better than discovering mid-incident that the fix needs a full reboot. A quieter failure comes from container memory limits: page-locked host memory used to stage transfers counts against the container’s host memory cgroup, so a container tuned by device memory and given a small host memory limit gets killed by the kernel out-of-memory killer during a perfectly normal transfer. The exit code says the container was killed for host memory while every GPU metric looks healthy, and the fix is a limit that accounts for pinned buffers and shared memory rather than for the model weights alone.