Using AI Agents to Teach Robots Like Microduck to Walk in Simulation

Nvidia published a step-by-step workflow for training a small open-source robot, Microduck from Pollen Robotics, to get through an obstacle course in simulation. Agents handle much of the setup. Hermes running Nemotron 3.5 Bolt on an RTX Spark turns a prompt into a plan, a concept image and an asset inventory. Astra then builds 3D models in FreeCAD and Blender and assembles the course in Isaac Sim. Isaac Lab and the Newton physics engine train the policy, and the result is validated back on the local machine.

What changed is less any single tool than the handoffs. Building simulation environments has been one of the slowest, most manual parts of robot learning. If agents can do a credible first pass, more small teams and hobbyists can train robots without a dedicated 3D art and sim engineering staff.

We think this is a genuinely useful direction, and the post is more careful than most agent demos. The prompts tell the agents to record provenance, note where objects in the concept image are cut off or hidden, log any dimensions they had to assume, and stay off physical hardware. Astra checks its cleaned-up reference images against the original crops. That's the habit we want to see from agents doing real work: show what's known and what's guessed.

The training numbers are concrete. A locomotion policy pretrained in Isaac Lab's existing Microduck environment got extra training on ramps and obstacles, using depth from a head camera plus robot state, across 4,096 parallel environments on a single RTX PRO 6000 Blackwell workstation GPU. That last detail matters. This isn't a data-center-only workflow.

Now the eyes-open part. There's a gap between the obstacle-course story and what's actually shown running on hardware. The sim training is described for ramps and uneven surfaces, and the physical clip is a locomotion policy walking and turning on a tile floor. Nvidia says plainly that hardware deployment is out of scope and the sim-to-real gap still needs work. We'd just caution readers not to blur the two.

The bigger risk if this scales is compounding assumptions. An image model invents a course, an agent guesses dimensions from a crop, another agent models hidden surfaces, and a policy learns from all of it. Each step is reasonable. Stacked together, it can produce a robot that is very good at a world that doesn't exist. Nvidia's mitigations, servo modeling for delay, backlash and voltage sag, plus domain randomization, are the right tools. But they only help if someone actually measures the real robot and feeds those numbers back in. Small robots falling over is a cheap lesson. Larger machines working near people won't be.

There's also a platform angle. Nearly every stage runs on Nvidia software and hardware, from the local box to the physics engine to the training framework. Isaac Lab and Newton are open source and Microduck is an open design, which helps. Still, the easiest path through this workflow is a single-vendor one.

Microduck is a small open-source robot from Pollen Robotics with public documentation and code. Isaac Lab already includes a Microduck example environment, and Nvidia has a separate sim-to-real tutorial for it. The SimReady Central Library mentioned in the post offers more than 2,000 free simulation-ready OpenUSD assets that can be mixed with generated ones.