How SWE-Serve Exposes the Gap Between Local Tests and Live Serving
Sep 23, 2026, 9:00 AM · NVIDIA Developer
NVIDIA’s SWE-Serve benchmark shows about one in three AI coding-agent patches that pass other checks still fail when a live inference server loads a real model and serves requests.
Why it matters
Coding agents are increasingly asked to touch inference stacks — the software that actually serves models. A green unit-test run is not the same as a server that loads weights and answers through its public API.
SWE-Serve makes that gap measurable on SGLang: 53 tasks from 83 merged PRs, and on the 19 tasks with live-serving checks, pass rates fall from 69.4% without those checks to 45.9% with the complete verifier.
From the desk
We’re covering this as an evaluation integrity story — and a warning shot for anyone shipping agent-written serving code.
The benchmark spans six families: speculative/advanced decoding, model enablement, kernels/quantization/performance, serving APIs, caching/runtime state, and distributed execution/scheduling. Twelve tasks run on CPU; 41 use a single H100. Median reference patches touch 553 lines across seven files. Nineteen tasks start a real server; three add a calibrated H100 performance gate.
The live-serving result is the headline. Across 627 patches on those 19 tasks, 147 flipped from fail to pass when live-serving tests were excluded. On Gemma 4 MoE alone, 16 of 33 patches cleared everything else and still failed at least one live-serving check — loading, expert routing, multimodal serving, batched generation with ordering and logprobs.
Cross-domain work is harder too. Tasks spanning more than one runtime domain (request I/O, scheduling, model execution, KV-cache) pass 21.3 points lower than single-domain tasks, and every tested model shows the same direction.
Model scores under mini-swe-agent closed-book conditions range from about 35% to 75% mean pass@1. Claude Opus 5 and GPT-5.6 Sol top the table near 75%, with very different cost and wall-time profiles. Cost does not map cleanly to quality — four models tied near 64% span roughly $0.95 to $7.24 per task.
Useful AI coding agents need this kind of hard mirror. Local green is not production green on serving systems. The harm if we ignore it is predictable: agents that look strong on repo benchmarks ship patches that break live inference, and teams learn the failure from customers. NVIDIA is careful: a SWE-Serve pass is not an SGLang maintainer endorsement or a merge-ready stamp.
I’m watching whether other engines (vLLM, TensorRT-LLM, etc.) get sibling suites, and whether agent harnesses start treating live-serving verifiers as non-optional rather than a nice-to-have.
Context
NVIDIA Developer blog by Jennifer Williams, Dave Farris, Jeff Farris, and Jiantao Jiao, Sep 23, 2026. SWE-Serve built with SGLang team input; closed-book evals blocked web retrieval of upstream solutions.
Who feels it
- AI coding-agent builders
- Live-serving verifiers should become first-class; unit-test-only scores will overstate readiness on inference code.
- Inference engine maintainers
- A public, PR-derived task suite raises the bar for what “agent can contribute here” claims mean.
- Enterprise ML platform teams
- Don’t let agents touch serving stacks without end-to-end load-and-serve gates on real hardware.
- Benchmark consumers
- Compare cost and wall time alongside pass@1 — similar scores hide very different operating economics.
What to watch
- Leaderboard movement as new models and harnesses submit closed-book runs.
- Whether multi-GPU and multi-node tasks appear in a later release.
- Sibling benchmarks for other serving engines beyond SGLang.
- Agent products that advertise SWE-Serve scores with live-serving included, not excluded.