TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor

Agents aren’t single-prompt chatbots. They call tools, read results, and keep reasoning across a growing conversation—exactly the pattern moving into robots, vehicles, and other edge boxes that can’t lean on a data-center GPU farm.

MLPerf Inference v6.1’s Edge Agentic benchmark finally measures that loop: multi-turn tool use, long shared context, and valid function calls under edge constraints. NVIDIA’s submission puts a concrete number on whether those workloads can run locally without collapsing into latency soup.

If edge agents become normal, who owns the inference stack—and how much power and memory they burn—starts deciding which products ship.

We’re reading this as an infrastructure milestone for useful on-device agents, not a cloud victory lap.

On a single Jetson AGX Thor Developer Kit, TensorRT Edge-LLM ran Qwen3.6-27B at 52.33 tokens per second and finished all 1,007 performance turns in 24 minutes and 36 seconds—6.4x faster than the published llama.cpp reference on the same kit (2 hours 37 minutes). Median time to first token was 247.12 ms; BFCL overall accuracy came in at 87.94%. Those are the scoreboard facts.

The engineering story underneath matters more than the headline multiplier. NVFP4 for weights and activations plus FP8 KV cache shrinks the memory wall that usually kills low-batch edge decoding. KV cache and recurrent-state reuse served roughly 96% of prompt tokens from hot cache across a trajectory that grows toward about 23.5K tokens—only ~0.5M of 13.6M prompt tokens needed a fresh prefill. Tree-based multi-token prediction (8 draft steps, top-2, 16-node tree) added about another 40% decoding gain over linear MTP on this function-calling load.

That’s the combo that makes long-horizon agents plausible on a 128 GB unified-memory box in MAXN mode: don’t recompute the past, and don’t generate one token per forward pass when the tool-call grammar is predictable.

Useful AI here is local capability—privacy, latency, offline operation—without pretending the cloud disappears. The downside if this stack becomes default: hardware lock-in and a performance gap that leaves open-source reference stacks looking slow by construction. Benchmarks also measure a harnessed workload; real robots and fleets will stress tool APIs, sensor noise, and power envelopes MLPerf doesn’t fully capture.

I’m watching whether developers actually ship from the release/0.9.1-mlpinf branch and the calibrated NVFP4 checkpoint—or whether the 6.4x stays a paper result for marketing decks.

MLPerf Edge Agentic splits into a performance phase (20 conversations, 1,007 turns, IoU inline accuracy) and an accuracy phase on BFCL v4 single-turn prompts with reasoning off. NVIDIA published the write-up September 16, 2026, with reproduction steps on the TensorRT Edge-LLM branch and a Hugging Face NVFP4 checkpoint.