ToyBlocks: A controlled world for robot learning and evaluation

Robot learning has a measurement problem. Labs show laundry folding and warehouse picks, then struggle to explain why a policy fails when the lighting changes or a block moves six inches. Without controlled worlds, progress looks like highlight reels.

Allen Institute for AI is releasing ToyBlocks, the first of what it calls FoundationWorlds: a narrow simulated tabletop with a robot arm, blocks that carry letters and pictures, automatic task and demonstration generation, and metrics beyond binary success. The point is to vary training data and model design, re-run the test, and trace the failure.

If robotics is going to graduate from demos to dependable systems, this kind of instrumentation is how you get there — not another viral clip of a humanoid making a bed.

We're glad someone is building the boring infrastructure. ToyBlocks is deliberately simple — push, flip, stack, spell with lettered blocks — and that constraint is the feature. Open-ended kitchens are too messy to isolate whether the planner, the low-level policy, or the perception stack failed. Here you can.

The generation stack matters as much as the world. Skills are trained with RL and motion planning in NVIDIA Isaac Lab, sometimes with privileged simulator state a real robot would not have, then sequenced by a task-and-motion planner into full demonstrations. Ai2 says it can run hundreds of plans in parallel on one GPU at up to 24× real time. The initial release includes 85,336 long-horizon demonstrations — roughly 1,600 hours and 144 million frames — plus trained Molmo 2 and ManiFlow showcases, assets, and eval code.

Early results are sobering in a useful way. Popular vision-language-action models still generalize poorly on controlled spatial tests even after large-scale pretraining; Ai2 improves them with techniques including LoRA and argues generalization remains unsolved. Hierarchical setups with Molmo 2 planning and ManiFlow executing do better on multi-step spelling tasks when letters start visible, and worse when they are hidden. Frontier agents via MCP tools can solve classical stacking from text, but search-and-spell with look-alike letters and infeasible scenes only clears about 63 percent — mostly perceptual mistakes, not planning collapse. World-model rollouts from Wan 2.2, when added as synthetic data, jump a policy's success in an unseen region from 19 percent to 78 percent in their tests.

Sim-to-real is the credibility check. ManiFlow atomic skills hit about 80 percent in sim and 81 percent on a Franka FR3; the hierarchical system hit 55.5 percent in sim and 50 percent across 30 real scenes. That transfer is encouraging and still far from a product.

I'm watching this as a bet that useful robot AI comes from scientific control, not bigger humanoids alone. The downside if the field ignores it: more demos, same brittle policies, and households that never get a robot that recovers when a sock falls behind the dryer.

ToyBlocks builds on Ai2's MolmoAct 2, MolmoBot, and MolmoSpaces work. A larger dataset, more skills, and fuller RL integration are promised in weeks. Future directions include bimanual manipulation, dexterous hands with force sensing, and unstructured pile play.