Configure, Certify, and Operate Kubernetes GPU Clusters with NVIDIA DSX OS

One slow GPU can stall a distributed training job while looking fine on basic health checks. Clusters drift after driver patches, node swaps, and capacity adds, and the original certification stops matching reality.

DSX OS, the open-source software layer of NVIDIA's DSX AI Factory Platform, ships three Apache-2.0 projects: AI Cluster Runtime (AICR) for version-locked configuration recipes rendered to Argo CD, Flux, or Helm; Cluster Readiness Engine (NVCRE) for NCCL and training certifications with performance thresholds and fault bisection; and NVSentinel for continuous GPU/node/network monitoring with optional cordon, drain, remediate, and revalidate loops.

Goodput — successful work completed — becomes the operating metric, not vanity utilization.

We're glad someone is productizing the boring reliability layer. AI factories fail quietly: a thermally soft GPU, a fabric that passes ping but fails all-reduce, a node that rejoins after repair without a rerun of the hard tests. DSX OS's configure / certify / operate loop is the right shape.

AICR pins versions, dependency graphs, and health assertions that look past CrashLoopBackOff-as-Running lies, then emits bundles your existing GitOps tool applies. NVCRE runs NCCL variants, DCGM level-4, and NeMo pretrain-style workloads with thresholds you set — bandwidth gates that catch the GPU just under target that never throws an Xid. Bisecting failing groups beats hand-quarantining 64 nodes. NVSentinel starts in monitor-only mode, publishes conditions, and only cordons when you trust the signals, with circuit breakers on how many GPU nodes can be quarantined in a window.

I want operators to adopt this before the next multi-million-dollar training stall. The downside if NVIDIA-shaped tooling becomes the only credible path: lock-in dressed as open source, and teams that cannot run non-NVIDIA accelerators through the same readiness culture. Forks and contributions will tell. Useful AI factories are measured in completed jobs per megawatt, not GPUs that look busy.

The post walks a cluster from recipe to certification to fault loop, including lifecycle ValidationRequests when nodes join or return from remediation. In-place driver and fabric upgrades still need operators to trigger validation explicitly. A KubeCon workshop and booth are part of the launch push.