Automating eval design and hillclimbing with Claude
**Playbooks** article by **Lance Martin**, published on the claude.dev blog on **September 28, 2026** (12-minute read, nine figures). It presents two subcommands added to Claude Code's **claude-api** skill, `/claude-api build-eval` and `/claude-api hillclimb`, after laying out the principles they apply. Three contributions. **(A)** Four properties of a good evaluation: tasks representative of production, scores that rise with the model and the effort, headroom below **100%** at the top, low variance from one run to the next; plus *adversarial sampling*: a hard case is kept because a human can say why, not because today's model fails it. **(B)** Disciplined hillclimbing: one change per round, separate **train/test** set, patch reverted if train rises while test stays flat. **(C)** Two measured examples: a customer-support benchmark (**44** tickets) whose cost drops from about **4.6 to 1 cent** per ticket, and the evaluation of the claude-api skill itself, from **66%** to about **88%**. A line from the text: *"it's important to read a sample of scored transcripts before trusting your evaluator."* Related to the author's piece on prompt caching, [[lancemartin-anthropic-prompt-auto-caching-claude-2026-02]].
Lance Martin (Anthropic) · blog claude.dev · avec remerciements à Misha Khalman pour le développement de la skill.