Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes

Most agent failures in the real world are not the model being dumb. They are the agent calling the wrong tool, skipping a step, or doing something nobody authorized. NVIDIA's new tutorial argues that the fix usually lives in the harness around the model, meaning prompts, tool descriptions, routing and code, and that generic benchmark scores won't tell a team where to look.

The worked example is pointed. An enterprise assistant asked to draft a message with project updates drafted it and then sent it without asking. That is the kind of mistake that turns a helpful agent into a liability, and the post walks through catching it in recorded traces, turning it into a repeatable test, and closing it with a plain-language instruction change.

We like this, with caveats. The core idea is sound engineering: treat agent failures like bugs, reproduce them with an evaluation, fix them, and accept a change only when it improves both the development tests and a held-out set the fixing agent never saw. That last part matters. Without held-out tasks, a coding agent tuning another agent will happily overfit to the test it was handed.

The send-without-confirmation example is the one I'd circle. The fix NVIDIA shows draws a clear line: a request to draft authorizes preparation only, and a token from a tool is a capability, not consent. That is exactly the distinction more agent builders need to write down. Consent should not be inferred from the fact that a button is reachable.

The engineering example with Dassault Systèmes' LEO assistant is the strongest evidence in the post. Malformed tool calls fell from 142 to zero after a validation step was added, and an overnight run of roughly 12 hours reported a 28% relative improvement in the overall evaluation score with 35% fewer tool calls across six scenarios. Those are meaningful gains if they hold up.

But read the source for what it is: a vendor tutorial on NVIDIA's own blog, promoting NVIDIA tooling, with results reported by NVIDIA. The claim that NeMo Compass caught anomalies a coding agent missed rests on internal benchmarks we can't inspect. The workplace demo uses 42 traces from a fictional company and a deliberately small evaluation set.

The bigger trajectory is agents optimizing agents, running loops overnight without a human in them. That can be genuinely useful, and it also raises the question of who reviews what the loop decided to change. We'd want every accepted harness change logged and reviewable, because an automated fix that quietly loosens a safety rule is still a regression, even if the score went up.

NeMo Compass analyzes stored agent traces to surface recurring problems and link findings to specific trace IDs. NeMo Eval Author gives coding agents such as Codex or Claude Code skills to write custom evaluations in Harbor or NeMo Gym formats. Both sit inside NVIDIA's NemoClaw blueprint for agent optimization, which runs a Hermes agent inside an OpenShell sandbox.