# narreddi-training-ai-paint-with-code-2026-03-01

## Veille

Case study by **Surya Narreddi** (with **Cameron Franz** and **Alex Wang**) on his personal website, dated **March 2026** (day unspecified; ~1,500 words), titled *Training AI to Paint with Code*. The authors train a language model (**Qwen 3.5 35B**) via reinforcement learning (**GRPO**) to produce watercolor paintings as **p5.brush** JavaScript code, with the code itself serving as the editable artifact. The underlying question: how to do RL when the reward — aesthetic — is not verifiable.

## Titre Article

Training AI to Paint with Code (RLing Qwen to paint with code)

## Date

2026-03-01

## URL

https://surya.website/rling-qwen-to-paint-with-code

## Keywords

reinforcement learning, GRPO, reward function, pairwise judge, subjective reward, creative tasks, Qwen, p5.brush, watercolor, code generation, HPSv3, GEPA, prompt optimization, reference pool, API hallucination, reward plateau, correlated signals, RLHF, Puppeteer, reward design

## Authors

Surya Narreddi — designer et développeur, auteur du projet (site personnel) ; collaborateurs Cameron Franz (infrastructure d'entraînement) et Alex Wang.

## Ton

First-person project narrative, halfway between a design memoir and research notes. The text proceeds through failure and then diagnosis: the plateau, reading the sub-rewards "in isolation," the fix. Choices are explained concretely, without unnecessary jargon, and limitations are stated plainly (all images in the pool are model-generated, the RLHF step "that we did not reach," the method being slower than conventional image generation). The conclusion defends the intent: participating in creation beyond the prompt, by acting on the model and on the artifact.

## Pense-betes

- **The problem**: an image generated by a model can only be edited by reprompting; generating **code** makes the artifact finely editable. RL requires a reward, and aesthetics is neither right nor wrong.
- **Diagnosis of the first grid**: five signals (recognizability, aesthetics, technique, depth, prompt adherence) correlated at 0.85-0.95 were "measuring the same thing five times over"; length, about a third of the reward, stopped giving any gradient after step 30; the only signal with real variance, **HPSv3**, carried a weight of 0.10.
- **Two fixes**: **pairwise** judging (the judge compares the render to two references; the reward is the win proportion) in place of a 0-to-10 score compressed near zero; a hand-rated **reference pool**.
- **Pool**: 581 references (117 "love," 266 "okay," 198 added to fill in poorly covered colors), all produced by models (AutoResearch with Opus 4.6, GPT-5.4, and Gemini 3.1 Pro; a batch on Gemini 3.1 Pro), for lack of human examples for this niche tool.
- **System prompt via GEPA**: a 400-line API reference caused the model to invent methods; the version with eight allowed methods, with no documentation or examples, produced usable output in three generations out of three for the first time.
- ⚠️ **Limitations**: the proposal to train a small reward model on the ratings (**RLHF** proper) was not carried out; a final training pass is announced, with a technical report in June 2026.

## RésuméDe400mots

**Surya Narreddi**, with **Cameron Franz** and **Alex Wang**, describes in **March 2026** a project to train a language model to paint with code. The starting point is an observation: with an image model, the only way to participate is through the prompt, and the image cannot be edited directly. Here, the model writes a **p5.brush** program in JavaScript that renders a watercolor; the code is the artifact, editable more finely than a prompt. The underlying question is doing reinforcement learning on creative tasks: RL works when the reward is verifiable, which aesthetics is not.

The loop is repeated thousands of times. The model (**Qwen 3.5 35B**) receives a prompt — for example, a peach hibiscus in watercolor — and writes the complete sketch; this is rendered as a PNG in an isolated Puppeteer environment; a judge model compares the render to two references drawn from a hand-rated pool; the judgment becomes a reward; **GRPO** updates the model.

The first reward grid had nine signals: a compilation gate, **p5.brush** usage, a length ramp around 3,000 tokens, **HPSv3** (a human-preference model), prompt adherence judged by GPT-5.4 and Gemini, and four quality judges. The model plateaued at 0.65, with all outputs being flat, clip-art-style flowers. Examining the sub-rewards in isolation showed that five judges were correlated between 0.85 and 0.95, that length (a third of the total) had saturated by step 30, and that **HPSv3**, the only signal with real variance, carried a weight of just 0.10.

The fix: **pairwise** judging (the judge picks the better watercolor between the render and two references; the reward is the fraction of comparisons won), which spreads scores out more than a compressed absolute score, plus a **reference pool** of 581 images drawn from 1,664 generations rated "love," "okay," or "nope" (117 "love"). The grid is reduced to four components: compilation and brush-usage gate (0.05), length (0.05), **HPSv3** (0.30), pairwise judge (0.60). With the same model and the same data, the old plateau is reached three times faster, progress continues, and the code shrinks from 13,500 to under 2,000 tokens. All images in the pool are model outputs, for lack of human examples.

The system prompt was also reworked with **GEPA**: a 400-line API reference caused the model to invent nonexistent methods; after 200 iterations against a seven-example judge, the best version turned out to be a closed list of eight methods, with no documentation. The author concludes that long references in a prompt favor hallucination and that a short, sharply bounded list constrains the model better.

In conclusion, the author writes that subjective RL requires hand-crafting the reward: too specific, and the model copies the examples; too loose, and it learns nothing. He does not present the method as a better way to produce images, but rather as a way to act on the prompt, the model, and the artifact. A final training pass and a technical report are announced for June 2026.

## GrapheDeConnaissance

- Surya Narreddi —publie→ article Training AI to Paint with Code (DOCUMENT, 0.97)
- Surya Narreddi —collabore_avec→ Cameron Franz (PERSONNE, 0.95)
- projet Paint with Code —utilise→ Qwen 3.5 35B (TECHNOLOGIE, 0.93)
- projet Paint with Code —utilise→ GRPO (METHODOLOGIE, 0.97)
- projet Paint with Code —utilise→ p5.brush (TECHNOLOGIE, 0.97)
- jugement par paires —résout→ plateau de récompense (CONCEPT, 0.92)
- jugement par paires —améliore→ grille de récompense (CONCEPT, 0.9)
- jugement par paires —utilise→ pool de références (CONCEPT, 0.93)
- grille de récompense —mesure→ plateau à 0,65 avec neuf signaux, juges corrélés à 0,85-0,95 (MESURE, 0.95)
- grille de récompense —mesure→ code réduit de 13 500 à moins de 2 000 jetons (MESURE, 0.93)
- GEPA —réduit→ hallucination d'API (CONCEPT, 0.9)
- Surya Narreddi —affirme_que→ une documentation d'API longue dans le prompt système favorise l'hallucination (AFFIRMATION, 0.88)
- Surya Narreddi —affirme_que→ le RL sur tâches créatives est un problème de design de la récompense (AFFIRMATION, 0.92)
- projet Paint with Code —s_oppose_à→ RLVR (METHODOLOGIE, 0.75)
- HPSv3 —fait_partie_de→ grille de récompense (CONCEPT, 0.93)

---
Canonical: https://www.thekb.eu/en/fiches/narreddi-training-ai-paint-with-code-2026-03-01/