Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo

NVIDIA Dynamo's preview keeps a fully initialized standby engine on the same GPUs as the active process. A GPU Memory Service owns weight allocations independently, allowing both engines to map one physical copy in high-bandwidth memory. The parked shadow retains its CUDA context, graphs, communicators, and weight mappings but no materialized key-value cache.

In NVIDIA's two-worker GLM-5.2 test on B200 nodes, a killed worker returned in 7.3 seconds with shadow recovery versus 283 seconds after a cold restart. Median post-fault time to first token was 1,311 milliseconds rather than 23,815 milliseconds, and only one of 398 requests exceeded five seconds versus 201 of 399 in the baseline.

This is an NVIDIA technical post benchmarking an NVIDIA preview on NVIDIA hardware, so the magnitude should be validated under independent workloads. Still, the mechanism is technically coherent: preserve expensive weight state outside the failed engine process and precompute process-specific initialization before failure rather than on the serving path.

The design makes an important distinction between process faults and infrastructure faults. It addresses crashes, recoverable CUDA errors, and hung engines when the GPU, driver, and node remain healthy. Hardware, node, and multi-node failures still require ordinary rescheduling, so shadow recovery complements rather than replaces fleet redundancy.

Signal Desk's read: the 39-fold recovery result is compelling, but the larger win is limiting overload on surviving workers. A cold restart shifts all traffic onto reduced capacity for minutes, degrading latency and decode speed. Fast local promotion can prevent one recoverable software fault from becoming a fleet-wide service-quality event.

The missing cache state is the main residual cost. A promoted shadow starts with an empty KV cache, creating a post-cutover latency bump. NVIDIA is working on transferring both cache memory and the prefix-cache index. That capability could improve continuity, but it will also make correctness, ownership, and synchronization during failover more demanding.

Cold starts reload model weights, size the KV cache, compile or autotune kernels, establish communicators, and recapture CUDA graphs. The GPU Memory Service uses CUDA virtual memory management so physical allocations can outlive one engine's CUDA context and be mapped into another process.

The preview requires Kubernetes 1.34 or newer with Dynamic Resource Allocation enabled and NVIDIA's GPU DRA driver. vLLM is the primary supported backend. Each worker pod contains two engine containers, a per-GPU memory sidecar, and a shared file lock that elects the active engine.