Surya Narreddi, with Cameron Franz and Alex Wang, describes in March 2026 a project to train a language model to paint with code. The starting point is an observation: with an image model, the only way to participate is through the prompt, and the image cannot be edited directly. Here, the model writes a p5.brush program in JavaScript that renders a watercolor; the code is the artifact, editable more finely than a prompt. The underlying question is doing reinforcement learning on creative tasks: RL works when the reward is verifiable, which aesthetics is not.
The loop is repeated thousands of times. The model (Qwen 3.5 35B) receives a prompt — for example, a peach hibiscus in watercolor — and writes the complete sketch; this is rendered as a PNG in an isolated Puppeteer environment; a judge model compares the render to two references drawn from a hand-rated pool; the judgment becomes a reward; GRPO updates the model.
The first reward grid had nine signals: a compilation gate, p5.brush usage, a length ramp around 3,000 tokens, HPSv3 (a human-preference model), prompt adherence judged by GPT-5.4 and Gemini, and four quality judges. The model plateaued at 0.65, with all outputs being flat, clip-art-style flowers. Examining the sub-rewards in isolation showed that five judges were correlated between 0.85 and 0.95, that length (a third of the total) had saturated by step 30, and that HPSv3, the only signal with real variance, carried a weight of just 0.10.
The fix: pairwise judging (the judge picks the better watercolor between the render and two references; the reward is the fraction of comparisons won), which spreads scores out more than a compressed absolute score, plus a reference pool of 581 images drawn from 1,664 generations rated "love," "okay," or "nope" (117 "love"). The grid is reduced to four components: compilation and brush-usage gate (0.05), length (0.05), HPSv3 (0.30), pairwise judge (0.60). With the same model and the same data, the old plateau is reached three times faster, progress continues, and the code shrinks from 13,500 to under 2,000 tokens. All images in the pool are model outputs, for lack of human examples.
The system prompt was also reworked with GEPA: a 400-line API reference caused the model to invent nonexistent methods; after 200 iterations against a seven-example judge, the best version turned out to be a closed list of eight methods, with no documentation. The author concludes that long references in a prompt favor hallucination and that a short, sharply bounded list constrains the model better.
In conclusion, the author writes that subjective RL requires hand-crafting the reward: too specific, and the model copies the examples; too loose, and it learns nothing. He does not present the method as a better way to produce images, but rather as a way to act on the prompt, the model, and the artifact. A final training pass and a technical report are announced for June 2026.