Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4

Plenty of AI workloads only need part of a GPU, but clusters often hand out whole devices anyway, leaving capacity idle. Older sharing approaches each came with a catch. Time-slicing let processes take turns without enforcing memory, so one greedy job could starve its neighbors.

CUDA 13.4 adds a memory boundary that applications do not need to know about. A workload can ask for an exact amount, or a guaranteed minimum with room to grow, and an allocation beyond its limit fails just as it would on a smaller card. A new open-source Kubernetes operator, GPU Fractioning, brings that enforcement into clusters.

This is infrastructure news, and it is the useful kind. The operator runs alongside the NVIDIA GPU Operator, sets up Multi-Process Service on each GPU node, applies each container's memory limits and compute mode before it starts, and reports usage per fraction instead of per whole GPU. KAI Scheduler, starting with version 0.18.0, is the first scheduler wired up to it through a mode called NvFractions. The design is open: any scheduler that writes the right Pod annotations can use the enforcement.

We think the real win is predictability. Teams running notebooks, small fine-tuning jobs, and inference services side by side have long had to choose between wasting hardware and risking noisy neighbors. Enforced memory limits, plus a choice between time-slicing and SM-sharing for compute, give platform teams a clearer contract to offer their users.

The trade-offs are spelled out, and they matter. In SM-sharing mode, workloads run in parallel but share a single error domain, so one crash can affect others on the same GPU. NVIDIA's own guidance is to reserve that mode for workloads that trust each other. Time-slicing keeps each workload isolated but means taking turns. A single GPU can only run one mode at a time, which adds a placement constraint the scheduler has to respect.

I'm also watching the ecosystem angle. This tooling sits inside NVIDIA's own stack, from CUDA to the scheduler organization on GitHub. It is open source, which is good, but it also deepens how much of the cluster's behavior depends on one vendor's software. For most teams that is already the reality.

Our read: if GPU time is the scarcest resource in AI, tools that let organizations fit more work onto hardware they already own are among the most practical advances of the year. It will not make headlines, but it may save real money.

NVIDIA already offered several ways to share GPUs. Multi-Instance GPU splits a card into hardware-isolated slices. Time-slicing lets processes alternate on the whole GPU. Multi-Process Service lets processes run concurrently in separate address spaces. CUDA 13.4 adds transparent memory enforcement on top of these.

NVIDIA is also inviting the community to get involved at KubeCon + CloudNativeCon NA 2026 in Salt Lake City, Nov. 9 to 12.