Training a coding model to paint watercolours with TRL and OpenEnv

This Hugging Face blog reproduces Surya Narreddi's viral video — a language model writing JavaScript for p5.brush to paint watercolours, 1.5 million views — with every artifact published: environment, pool, scripts, models. The author uses TRL and OpenEnv, trains Qwen3.5-35B-A3B with LoRA on an H200, and restricts the model to ten library methods so the look stays watercolour. The reward mixes HPSv3, an open 7B preference model, with a pairwise judge (Qwen3-VL-30B) scored against a 178-image pool the author rated into 'love' and 'okay' tiers, all model-generated from iNaturalist hibiscus photos. No human-made paintings are in the pool.

Three runs: hps-only to prove learning, then hps-led and judge-led. After a stretch of flat curves, the unlocks were a higher learning rate, all-linear LoRA on a MoE that names layers unusually, and a constant-with-warmup schedule. All three runs learn; both judge runs were stopped at 110 of 200 planned steps, 15–18 minutes a step. Most of the gain is fewer bad paintings, not better best-of. With the judge on, paint coverage roughly doubles and the top of the distribution moves. About 1.5% of rollouts (5.2% in the worst run) were infra failures scoring as zeros until those paths returned None. An OpenEnv websocket bug was patched upstream.

This is a platform blog, and it earns the length. The intellectual claim is not that a 35B MoE can emit pretty JS. Frontier models already can. The claim is that a small, hand-rated set can steer GRPO toward one person's taste, and that HPSv3 alone makes paintings reliable without making them better. The judge term is what asks for more pigment.

The pool is the product. 178 hibiscus renders, no human watercolours, two tiers so a weak policy still gets wins. That is honest about the medium (p5.brush has no large human corpus) and a limitation. GRPO will collapse variety onto whatever the set contains. One flower in, one flower out.

Signal Desk's read: 'RL over taste' is the sentence worth stealing. Most post-training is math and unit tests because those rewards are cheap. This post is a reminder that aesthetic RL is mostly curation plus infra that does not lie to you. Scoring a timed-out render as 0.0 is how you train on outages. Publishing the rollouts, the pool, and the three mixes is how you make a viral art video into a method.

The model ignoring 'fifteen to thirty shapes' because the reward did not pay for it is the funniest rigor in the piece. Policies do what you grade.

Narreddi's original blog and thesis came from the art side; this is the engineering reproduction with open weights. It sits in a lineage the author names — DeepDream, Ridler's tulips — where the dataset is part of the work. Hugging Face's TRL and OpenEnv are the scaffolding being demonstrated.