AI · Oct 28, 2026
RTX Spark laptops get open speech models and a dial for background agentsInside NVIDIA’s 10x agentic inference claim: read the footnote
· Updated · Signal Desk newsroom
Analysis of reporting by NVIDIA Developer. How we work

NVIDIA’s technical post on MLPerf Endpoints v1.0 explains how Vera Rubin serves multi-turn agents more efficiently. The footnote shows a provisional result compared at very different concurrency levels, which shapes what the headline means.
Key takeaways
- MLPerf Endpoints v1.0 tests agentic coding and enterprise workflows via API endpoints and checks tool calls and intent against reference values.
- NVIDIA reports 5x to 10x higher throughput per kilowatt and 3x to 5x higher interactivity versus GB300 NVL72 across three models.
- The 10x Kimi K3 figure compares a 32-accelerator Vera Rubin run at 1,024 concurrent queries with a 48-accelerator GB300 run at 48.
- Vera Rubin results are provisional and pending MLCommons verification; the GB300 results are verified.
- Gains come from HBM4, sixth-generation NVLink, prefix caching, Dynamo routing, disaggregated serving and model-specific kernels.
A benchmark built for agents
MLPerf Endpoints v1.0 is a welcome addition because it tests what agent workloads actually look like. Coding sessions accumulate history over many turns; enterprise workflows reuse shared instructions, tool definitions and business rules. The benchmark serves these through API endpoints, checks generated tool calls and workflow intent against reference values during performance runs, and separately evaluates task-solving on SWE-bench Verified. That last part matters: faster serving that degrades answers is not faster serving.
NVIDIA’s submissions covered Kimi K3, DeepSeek-V4.1-Flash and Qwen3.6-35B-A3B. Against verified GB300 NVL72 results, it reports 5x to 10x higher throughput per provisioned kilowatt at a shared minimum responsiveness of 50 tokens per second per user, and 3x to 5x higher interactivity at matched concurrency. The companion announcement rolled this into a broader MLPerf sweep claim.
What the footnote shows
The headline Kimi K3 comparison is spelled out in a footnote, and it is worth reading. The Vera Rubin NVL72 run used 32 accelerators at 1,024 concurrent queries, producing 41,414 tokens per second at about 100.9 kW. The GB300 NVL72 run used 48 accelerators at 48 concurrent queries, producing 2,210 tokens per second at 60 kW. By our arithmetic from those figures, that is roughly 410 versus 37 tokens per second per kilowatt, about 11x. Both runs clear the 50-tokens-per-second interactivity floor, Vera Rubin at 50.4 and GB300 at 56.2.
So the comparison is legitimate by the stated rule, best throughput per kilowatt at a minimum responsiveness, but it pits Vera Rubin at very high load against GB300 at low load. Time to first token at the 90th percentile was 4.7 seconds for the Vera Rubin run versus 2.3 seconds for GB300. And the Vera Rubin results are provisional, subject to change pending MLCommons peer review. None of that erases a large generational gain; NVIDIA’s step-function chart deliberately avoids interpolating between measured points, which is good practice. It does mean the 10x is a ceiling under specific conditions, not a typical uplift.
Where the gains come from
NVIDIA attributes the results to hardware and serving software working together. Rubin GPUs pair HBM4 memory for fast access to weights and conversation state with Tensor Cores and sixth-generation NVLink. On the software side, prefix caching and state reuse avoid recomputing shared history, Dynamo routing balances load, and disaggregated serving lets prompt processing and token generation be tuned separately. Model-specific work includes DeepSeek’s MegaMoE kernel and Qwen’s multi-token prediction verified on Tensor Cores.
The software story is the transferable part. Prefix reuse and disaggregation help any agent deployment where context grows across turns, and these techniques are not unique to Rubin. Operators deciding on capacity should benchmark their own agent traces, because the ratio of shared context to fresh tokens drives how much of NVIDIA’s gain they will see. The same theme runs through NVIDIA’s own memory-codesign arguments.
Who it affects
- Inference operators
- Prefix caching, routing and disaggregation offer gains worth adopting regardless of GPU generation.
- Capacity planners
- Throughput per kilowatt at a responsiveness floor is the right metric, but test at your own concurrency.
- Benchmark watchers
- MLPerf Endpoints adds correctness checks that make agentic speed claims more credible.
What to watch
- MLCommons verification of the provisional Vera Rubin results.
- Submissions from other hardware vendors to MLPerf Endpoints.
- Comparisons at matched concurrency and time-to-first-token targets.
FAQ
What does MLPerf Endpoints measure?
Serving capacity and responsiveness on agentic coding and enterprise workflows via API endpoints, with tool calls and intent checked against references.
Is the 10x figure verified?
Not yet. The Vera Rubin results are provisional pending MLCommons verification; the GB300 comparison results are verified.
Were the systems tested under the same load?
No. The footnoted Kimi K3 comparison uses 1,024 concurrent queries for Vera Rubin and 48 for GB300, both above a 50 tokens-per-second interactivity floor.
Companies: NVIDIA
More on NVIDIA
AI · Nov 5, 2026
NVIDIA’s four open Cosmos 3 Nano models let robot and driving teams simulate worlds that reactAI · Nov 16, 2026
Rubin HPC is Nvidia’s bet that science agents will be bottlenecked by simulation, not reasoningAI · Nov 9, 2026
Nvidia’s Korean startup bets span interconnect, open models, robot hands and document AIAI · Nov 8, 2026
Nvidia’s Dynamo recipes speed four Korean open models to production, tuned for B200s