Benchmarking LLM Inference at Scale with AIPerf

NVIDIA’s developer blog introduces AIPerf as the designated successor to GenAI-Perf—a ground-up rewrite that does not sit on Perf Analyzer. The stated problem: hand-rolled curl scripts and single-process Python load generators hit GIL limits and produce numbers you cannot trust.

AIPerf uses worker processes for load, separate record-processor services for results, and ZMQ coordination so the client does not become the bottleneck under high concurrency. It supports 15+ endpoint types (chat, responses, NIM rankings, image generation, and more), public datasets such as ShareGPT, and trace replay formats from Mooncake, Baseten, and WEKA AgentX.

Engineers can shape arrivals with constant, Poisson, and gamma patterns, tunable burstiness, and ramps on concurrency and request rate. Core metrics include TTFT, inter-token latency, request latency, and output token throughput with percentile breakdowns; GPU power, utilization, and memory can attach when DCGM or pynvml is available. A walkthrough uses Qwen3-0.6B on vLLM with streaming required for TTFT/ITL measurement.

We’re for tooling that makes “is this fast?” an honest question—because vibes-based latency is how bad deploys ship.

Replacing GenAI-Perf with a multiprocess design is the right architectural lesson: if your client saturates first, you are profiling Python, not the server. Endpoint breadth and production trace replay matter more than another synthetic 128/128 brag sheet. Percentiles plus GPU telemetry in one run is how you catch the p99 that averages hide.

Useful AI inference is an operations discipline. AIPerf will not fix a bad serving stack, but it removes the excuse that measurement was too hard. The downside of better benches is marketing misuse—same as every other leaderboard—so publish configs and seeds with the numbers.

I’m watching adoption beyond NVIDIA’s stack and whether teams treat Poisson/production-trace runs as the default, not the static demos.

NVIDIA published the technical blog on September 18, 2026, with authors including Francesco Di Natale, Elias Bermudez, Anthony Casagrande, Matthew Kotila, Harshini Komali, and Ganesh Kudleppanavar.