On Kubernetes, GPU time slicing is not really a hardware feature you turn on — it is a change to what the device plugin tells the scheduler. Configure a replica count and a node with one physical accelerator starts reporting several allocatable GPU units, so several pods can land on it instead of one. Nothing about the silicon changed — the GPU still interleaves work from every process that holds a context on it. What changed is that the cluster now believes it has more capacity than it has, and it will happily schedule against that belief. That single fact drives everything an operator needs: how to choose a ratio, why out-of-memory becomes your pager’s favourite alert, why latency-bound serving has no business on a time-sliced node, and when to move to hardware partitioning instead.
What the device plugin actually advertises
Kubernetes only understands whole, countable resources. A GPU is exposed as an extended resource — an integer a node reports as allocatable and a pod requests. Enabling time slicing does not add a fractional resource type; it multiplies that integer. A node that reported one unit now reports the configured replica count, and the scheduler treats those units as separate cards.
The replicas are pure bookkeeping. Every pod that gets one is handed the same physical device, opens its own CUDA context on it, and shares the hardware with every other tenant on that node. The plugin is misreporting capacity to the scheduler on purpose, because misreporting is the only way to express ‘this GPU is idle most of the time’ in a resource model with no concept of fractions or duty cycle.
What replicas do not buy you
Three kinds of isolation people assume they are getting, and none of them arrive with a replica count:
| Isolation | Time slicing gives you | Consequence |
|---|---|---|
| Memory | Nothing — one shared HBM pool, first come first served | Any tenant can starve every other tenant |
| Compute | Interleaving, but no guaranteed share or weight | Throughput per pod is whatever is left over |
| Fault | Nothing — one device, one failure domain | A device-level fault or reset takes out every co-tenant |
The fault row is the one that surprises teams in production. An uncorrectable memory error, a hung kernel that forces a device reset, or a driver-level fault is a property of the card, not of the container that triggered it. Every pod holding a replica of that card goes down together, and from the scheduler’s side they looked like independent workloads on independent resources. Your blast radius is the node, not the pod.
Choosing a ratio from duty cycle, not from wishful thinking
The only defensible input to the replica count is measured duty cycle: over a representative window, what fraction of wall-clock time does a typical tenant actually keep the GPU busy? A data-science notebook that runs a cell every few minutes might sit at a few percent. A CI job that builds for eight minutes and tests kernels for two might sit near twenty. As a starting point, a ratio somewhere near the inverse of that fraction keeps the device usefully loaded without making every tenant wait behind someone else’s work.
Then apply the ceiling duty cycle cannot see: memory. The sum of peak resident footprints across co-tenants must fit in the card’s HBM, because nothing will enforce it for you. In practice memory, not utilisation, caps the ratio: compute headroom might justify eight tenants, but four working sets that each want a quarter of the card mean the answer is four.
Out of memory is the dominant failure mode
On a time-sliced node, the GPU memory allocator is a shared, unpoliced resource with no per-pod quota. The first tenant to ask for memory gets it; the last one to arrive gets an allocation failure and crashes. The pod that dies is very often not the pod that caused the problem, which makes the failure look random and makes on-call triage miserable.
Framework defaults make this worse. Caching allocators grab memory and keep it, so a process that briefly needed a large buffer holds that reservation for its lifetime even while idle; some frameworks historically pre-allocated nearly the whole device at startup unless told to grow on demand. On a dedicated GPU that is good behaviour; on a shared one it is a land grab. A serious rollout therefore forces tenant images to enable incremental growth and cap their own per-process memory fraction — that self-restraint is the only enforcement mechanism that exists.
Latency variance, and why SLO serving is out
Sharing a GPU does not slow work down smoothly. It adds variance. Your request’s kernels are interleaved with whatever the co-tenants are running, so the same inference that completes in a few milliseconds on an idle node can take several times longer when a neighbour is mid-training-step. Median latency may barely move while the tail stretches badly, and the tail is what your error budget is written against.
There is no priority knob to fix this. The interleaving does not know that one pod is a customer-facing endpoint and another is a nightly evaluation sweep; there are no shares, no weights, no way to say ‘this tenant first.’ Nor is the effect predictable from the replica count, since it depends on what the neighbours happen to be doing. Any workload with a p99 target belongs on a dedicated device or a hardware partition. Time slicing trades tail latency for utilisation, and that trade is not negotiable per pod.
The workloads it is genuinely right for
Invert the argument and the sweet spot is obvious: anything with a low duty cycle, a small footprint, a human in the loop, and no latency contract. Interactive notebooks are the canonical case — a researcher spends most of the session reading, typing and thinking, and the GPU sits idle between cells. Dedicating a whole accelerator to that is the most expensive idling in the datacenter.
CI is the second strong case. Test suites that exercise GPU code paths need a real device but use it in short bursts, and queueing them behind whole-GPU allocations wrecks pipeline latency for no hardware reason. Development environments, teaching clusters, small-model experimentation, and batch jobs that tolerate a variable finish time fit the same profile. The common thread: these tenants care about availability of a device at all far more than the throughput they get from it — exactly the preference time slicing rewards.
Requests and limits stop meaning anything
For CPU and memory, Kubernetes gives you a request that drives scheduling and a limit that the kernel enforces at runtime. Extended resources have no such split: request and limit must be equal, and there is no runtime enforcement at all. A pod holding one GPU replica has no ceiling on what it can consume — the number in the manifest only decided where it landed.
A second trap follows from the same bookkeeping. Asking for several replicas does not reserve a proportionally larger share; it consumes several slots of a fictional capacity and yields the same unrestricted access to the same card. It reduces how many neighbours you have, not how much you get. And since a node advertises either real GPUs or sliced ones, mixing modes means separate node pools with distinct labels and scheduling rules that steer exclusive workloads away. Cluster topology, not pod spec, is where this decision lives.
Observability when many pods report one device
GPU telemetry is collected per device, so on a time-sliced node every utilisation and memory series describes the aggregate of all tenants. A dashboard that joins those metrics to pods by node will happily attribute the same ninety-percent utilisation to each of six pods, and a naive sum will report six hundred percent. Capacity planning built on that data is worthless.
Per-process attribution is the missing piece, and it is genuinely hard to get: containers run in their own PID namespaces, so the device-level process list frequently cannot be mapped back to pods without extra plumbing, and inside a container the usual tooling often shows no processes at all. Plan for this before rollout. Label per-device metrics with the replica count so whoever reads the graph knows how many tenants share the line, treat device utilisation as a node signal rather than a pod signal, and accept that per-pod GPU cost attribution here is an estimate, not a measurement.
Guardrails that make it survivable
Because the hardware enforces nothing, every control has to be administrative. Put time-sliced nodes in their own pool with their own labels and taints, so nothing lands there by accident and a bad ratio can be rolled back by draining one pool. Apply namespace quotas on the GPU extended resource so a single team cannot claim every slot on the shared fleet.
Set an eviction story: idle-culling for notebook servers, wall-clock deadlines for CI pods, preemption priorities so squatters lose to short interactive work. Bake memory-hygiene settings into the base images your tenants build from rather than a wiki nobody reads. And record the ratio’s justification beside the config, or the number gets treated as a tuning dial and quietly raised until the first OOM storm. Every one of these is policy, not mechanism — precisely the limitation you are accepting.
The migration path to MIG
The signals that you have outgrown time slicing are consistent: recurring out-of-memory kills you cannot attribute, tenants who need a latency guarantee, workloads from different trust boundaries on the same card, or a chargeback model that requires real per-tenant measurement. All four are isolation problems, and no replica count solves an isolation problem.
Hardware partitioning is the answer on parts that support it: instances get their own memory and their own slice of compute and cache, so a neighbour cannot take your bandwidth, and faults are far better contained. The cost is rigidity — fixed profiles, a node drain to reconfigure, and any capacity you carve out but do not use is wasted. Many clusters run both: partitioned nodes for anything with a contract, sliced nodes for the interactive and CI fleet, and, where supported, slicing layered on top of a partition to squeeze the idlest tenants further.