Topology-Aware Workload Scheduling with NVIDIA Topograph

AI factories waste money when tightly coupled GPU jobs sprawl across distant domains. Traffic hops shared links, latency climbs, and GPUs sit drawing power while waiting on data that should have been next door.

NVIDIA’s Topograph toolkit is meant to fix the missing map. It discovers cluster topology from cloud APIs or on-prem fabrics, normalizes it, and publishes it where Slurm and Kubernetes already listen — then regenerates that view when the cluster changes. Scheduling without current locality data has been the quiet tax on multi-node training and inference; this is an attempt to retire that tax.

We’re bullish on the problem statement. NVLink domains and Spectrum-X or InfiniBand fabrics are not abstract graphs — they’re the difference between a job finishing and a job thrashing. Fifth-gen NVLink on Blackwell-class systems is quoted here at 1.8 TB/s bidirectional per GPU; sixth-gen Vera Rubin at 3.6 TB/s. Quantum InfiniBand ports can hit 800 Gb/s. Schedulers that ignore those tiers burn tokens and watts for no good reason.

Topograph’s design is the part we like most: providers discover, engines publish. Cloud integrations already named include Google Cloud, Lambda, Nebius, Nscale, OCI, and Crusoe; on-prem paths cover InfiniBand via ibnetdiscover, Spectrum-X/NetQ, and Multi-Node NVLink domains. Output can be Kubernetes node labels, NFD resources, Slinky ConfigMaps, or Slurm tree/block/per-partition topology. Inside NVIDIA’s DSX OS story, it sits beside Dynamic Resource Allocation and KAI Scheduler for topology-aware gang scheduling — the mode that actually matters for multi-pod GPU work, not polite preferred affinity.

Useful AI infrastructure is often unglamorous plumbing. Continuously regenerating topology after node and pod events, aggregating duplicate generate requests, and giving operators a Helm path or a native Slurm package is the kind of work that compounds. The Slinky engine also matters because NVIDIA acquired SchedMD in December 2025; Slurm-on-Kubernetes is no longer a side hobby in this ecosystem.

The risk if this wins: operators assume labels equal ground truth. Topograph reflects reported topology, refreshed on generation and watched events — not every switch rewire. Cloud provider coverage is uneven by engine, and MNNVL support is narrower (DRA block topology only in the matrix they show). Wrong labels are worse than no labels because affinity and gang placement will confidently pack jobs into a fiction. I’m watching upstream contribution of new providers, how often labels lag reality in live factories, and whether KAI/Kueue adoption turns this from a blog demo into default cluster hygiene.

Sep 22, 2026 NVIDIA Developer Blog by Dmitry Shmulevich and Rob Esker. Topograph is open source under the dsx-ai-factory/topograph repo; the provider/engine matrix is described as current upstream main as of Sep 16, 2026.