Enabling Private High-Performance Production AI Inference with NVIDIA Confidential Computing

As LLM inference pulls in proprietary context, customer data, and regulated prompts, “just run it on a GPU” is no longer enough for a lot of enterprises. They need the model and the data protected while in use — not only at rest or in transit.

NVIDIA’s new engineering post argues Confidential Computing on Blackwell can deliver that without torching throughput. On a controlled DeepSeek-R1 workload across eight B200 GPUs, confidential mode kept more than 96% of baseline output-token throughput, with per-token latency overhead under 5%. That number, if it holds on real traffic, changes the buy-versus-wait calculus for private production inference.

We’re treating this as useful AI hardware work, not a press-kit miracle. The honest read is that confidential compute still changes the physics of the stack: host-to-device copies go through a software-encrypted bounce buffer because the GPU can’t touch protected CVM memory the usual way; pinned-memory assumptions break; CUDA-event timing gets jumpy enough to fool the autotuner; NVLink SHARP multicast isn’t available in the B200 CC config they tested. Leave TensorRT-LLM naive and you pay for security in stalls and bad tactic picks.

What NVIDIA is selling — and what we find persuasive — is co-design. TensorRT-LLM switches to CC-aware memory selection, moves token readback onto an async worker so decode doesn’t block, and times tactics with the GPU’s global timer under CC. Multi-GPU paths have to notice that NVLS isn’t there and pick collectives that don’t pretend multicast still works. Security and performance stop being separate tickets.

The benchmark design is also worth credit. They picked a long-context, long-generation, low-concurrency shape specifically to expose overhead instead of hiding it under load. Across concurrency 1–16, CC-on retained 96.1–98.2% of CC-off throughput and added 1.2–4.3% mean TPOT overhead on Intel TDX guests atop a DGX B200. That’s not “free,” and it’s not universal — one model, one framework build, one box. But it’s evidence that private inference doesn’t have to mean a 20–30% tax if the runtime is rebuilt for the trusted path.

The downside if this becomes normal: vendors and customers will treat “CC retained 96%” as a slogan without re-running the CC-on versus CC-off comparison on their own mix. Attestation, guest kernels, driver and VBIOS versions, and orchestration all sit in the critical path. I’m watching whether hyperscalers and regulated buyers publish their own retained-throughput numbers, and whether open inference stacks outside TensorRT-LLM copy these mitigations instead of bolting encryption on and hoping.

Lorna Aine’s Sep 22, 2026 NVIDIA Developer Blog post details TensorRT-LLM adaptations for NVIDIA Confidential Computing on Blackwell, including methodology tables for a DeepSeek-R1-0528-NVFP4 workload (32K in / 1K out, TP=8) on one DGX B200 with Intel TDX.