# lancemartin-anthropic-automating-eval-design-hillclimbing-2026-09-28

## Veille

**Playbooks** article by **Lance Martin**, published on the claude.dev blog on **September 28, 2026** (12-minute read, nine figures). It presents two subcommands added to Claude Code's **claude-api** skill, `/claude-api build-eval` and `/claude-api hillclimb`, after laying out the principles they apply. Three contributions. **(A)** Four properties of a good evaluation: tasks representative of production, scores that rise with the model and the effort, headroom below **100%** at the top, low variance from one run to the next; plus *adversarial sampling*: a hard case is kept because a human can say why, not because today's model fails it. **(B)** Disciplined hillclimbing: one change per round, separate **train/test** set, patch reverted if train rises while test stays flat. **(C)** Two measured examples: a customer-support benchmark (**44** tickets) whose cost drops from about **4.6 to 1 cent** per ticket, and the evaluation of the claude-api skill itself, from **66%** to about **88%**. A line from the text: *"it's important to read a sample of scored transcripts before trusting your evaluator."* Related to the author's piece on prompt caching, [[lancemartin-anthropic-prompt-auto-caching-claude-2026-02]].

## Titre Article

Automating eval design and hillclimbing with Claude

## Date

2026-09-28

## URL

https://claude.dev/blog/automating-eval-design-and-hillclimbing/

## Keywords

evaluation, eval design, hillclimbing, claude-api, build-eval, hillclimb, Claude Code, skill, adversarial sampling, train and test sets, evaluation overfitting, LLM-as-judge, programmatic grader, variance, confidence interval, headroom, cost reduction, effort, prompt caching, harness, reward hacking, Opus 5.5, Sonnet 5

## Authors

Lance Martin (Anthropic), blog claude.dev, avec remerciements à Misha Khalman pour le développement de la skill.

## Ton

**Profile**: a practical engineering guide, written by an Anthropic practitioner for teams building an application or skill on the Claude API. Audience: developers and quality leads for LLM-based systems.

**Style**: two symmetrical parts (evaluation design, then hillclimbing), each first laying out principles and then the command that applies them, followed by two measured examples. Numbered lists, schematic figures, definitions given along the way (harness, noise). The text presents its own tools as the implementation of the principles it states.

**Epistemic position**: the figures in both examples come from benchmarks internal to Anthropic (customer support, evaluation of the claude-api skill) and are not reproducible from the article. The reference result is the one from the held-out test set (90.5% versus 78.6%), with scores on the research set used for tuning.

## Pense-betes

- **Four marks of a good evaluation.** Tasks drawn from production; a score that rises with a stronger model and more effort; best model clearly below 100%; low variance. Warning sign: a task that fails on every run regardless of how many times it's repeated is probably ambiguous or poorly graded.
- **Criterion for a good task.** Two domain experts reach the same verdict, and everything the grader checks is stated in the task. Variance can also come from configuration (effort misapplied) or environment (residual state, git history that hands over the answer).
- **Adversarial sampling.** Choosing the cases the current model fails amounts to sampling the valleys of a single model. The cases kept are those judged difficult by a human, and those drawn from bugs or tickets. Raw user traffic can skew toward the easy, since users tend to attempt what they expect to succeed at.
- **`build-eval`, source order.** Production transcripts (after a question on retention and sensitive data), bug reports and tickets, five to ten hand-written cases, cases synthesized from the code. A generated page shows all inputs and waits for the user's confirmation.
- **Grader choice.** The cheapest one that fits: programmatic checking if outputs are constrained; otherwise LLM-as-judge with a checklist of verifiable claims (not a 1-to-5 scale), a judge model distinct from the model being tested, and a blind comparison in random order against a reference.
- **Diagnostic checks on the baseline.** Grader replayed twice on the same output; plumbing (timeouts, API errors, truncated responses); headroom, with a warning if the baseline exceeds about 95%, in which case targeting cost or latency instead.
- **Where to apply hillclimbing.** A surface that is cheap to change (prompts, skills rather than harness code), a score attributable to that surface (a skill's trigger rate and its description), a bounded objective. Cost is a robust target even on a saturated evaluation.
- **Three safeguards against overfitting.** Separate train and test; never paste the content of failures into the prompt; keep answers structurally out of the model's reach. Examples of leakage: an OCR tool added because the evaluation happens to benefit from it, or a public repository from which the harness retrieves the solution.
- **`hillclimb` loop.** Choice of the modifiable surface (system prompt, skills, tool descriptions, model and effort, harness code), an objective, a random train/test split, a prior check that noise is lower than the smallest useful gain. Each round reads the train transcripts and proposes a patch; kept if both train and test rise, reverted otherwise. Delivers the version that scores best on test, with confidence intervals; if the gain is within the noise, the recommendation is not to merge.
- **Reflection step.** When the score stalls for two or three rounds, no editing happens: each remaining failure is classified by cause. This step revealed skill content that was present but ignored, Claude writing outdated API forms drawn from its training (a fixed thinking budget instead of adaptive reasoning), and flawed tasks or graders.
- **Cost example (customer support).** 44 tickets, 30 for research and 14 held out for test. Starting point: Opus 4.8 at high effort, 74.4% accuracy, 4.6 cents per ticket. Prompt audit, then Opus 5.5 at low effort (87.8%, 1.9 cents), then Sonnet 5 at low effort (88.9%, 1 cent), finally an improved prompt (98.9%). On the 14 held-out tickets: 90.5% versus 78.6%, for about a fifth of the cost.
- **Performance example (claude-api skill).** From 66% to 74% by adding eight missing features, 77% after fixing the C# and Java type tables, 80% with a mapping table from old to new API forms, about 88% after fixing two flawed tasks and graders.

## RésuméDe400mots

Lance Martin, of Anthropic, publishes a guide on the claude.dev blog about designing evaluations and improving an application or skill against them without fooling yourself. He announces the addition of two subcommands to Claude Code's claude-api skill: `/claude-api build-eval` builds an evaluation in the repository, `/claude-api hillclimb` improves the application against it, one change at a time, with a held-out set of examples to detect overfitting.

The first part states four properties of a good evaluation. Tasks reflect production. Scores rise with a more capable model and higher effort; otherwise, ambiguous tasks or a miscalibrated grader are to blame. The best model remains clearly below 100%, without the gap being due to impossible tasks: a task that fails on every run is suspect, and a good task is one on which two experts reach the same verdict. Finally, variance is low, which requires a stable grader, effort applied consistently, and an environment free of residual state. The author adds a caution about adversarial sampling: choosing the cases the current model fails measures its failure footprint rather than real difficulty. He recommends cases judged difficult by a human, real failures drawn from tickets or bugs, while noting that raw traffic can skew toward the easy.

`build-eval` questions the user, then builds the evaluation with approval pauses. Inputs come, in order, from production transcripts, tickets, five to ten hand-written cases, then synthesized cases. The grader is the cheapest one that fits: programmatic when outputs are constrained, otherwise a judge model with a checklist of verifiable claims. Claude scores a few cases and asks whether the user would have judged them differently, then runs the baseline with a confidence interval and three checks: grader stability, plumbing, headroom.

The second part covers hillclimbing. It suits surfaces that are cheap to change, whose score is attributable, with a bounded objective; cost is a good one. Against overfitting, three safeguards: separate train and test, never paste failures into the prompt, keep answers out of reach. The command splits the evaluation, checks that noise is lower than the targeted gain, proposes one patch per round, and reverts it if train rises alone. On stagnation, it classifies failures by cause before editing.

Two examples illustrate all this. On a 44-ticket support benchmark, moving from Opus 4.8 to Opus 5.5 then to Sonnet 5 at low effort, with an improved prompt, yields 90.5% versus 78.6% on the held-out tickets, for about a fifth of the cost. On the claude-api skill, the evaluation rises from 66% to about 88%: missing sections, type tables, a table of deprecated API forms, and fixes to flawed tasks and graders.

## GrapheDeConnaissance

- Lance Martin —travaille_chez→ Anthropic (ORGANISATION, 0.95)
- Anthropic —publie→ article Automating eval design and hillclimbing (DOCUMENT, 0.95)
- claude-api —fait_partie_de→ Claude Code (TECHNOLOGIE, 0.93)
- claude-api build-eval —fait_partie_de→ claude-api (TECHNOLOGIE, 0.96)
- claude-api hillclimb —fait_partie_de→ claude-api (TECHNOLOGIE, 0.96)
- claude-api build-eval —utilise→ LLM-as-judge (METHODOLOGIE, 0.92)
- Échantillonnage adversarial —s_applique_à→ conception d'évaluation (METHODOLOGIE, 0.9)
- Lance Martin —recommande→ choisir les cas difficiles parce qu'un humain sait dire pourquoi ils le sont (AFFIRMATION, 0.93)
- Lance Martin —recommande→ lire un échantillon de transcriptions notées avant de croire l'évaluateur (AFFIRMATION, 0.92)
- Hillclimbing —utilise→ jeu train/test séparé (CONCEPT, 0.94)
- Jeu train/test séparé —résout→ surapprentissage de l'évaluation (CONCEPT, 0.9)
- Surapprentissage de l'évaluation —observé_dans→ outil d'OCR ajouté au harnais parce que l'évaluation en profite (AFFIRMATION, 0.88)
- claude-api hillclimb —recommande→ annuler un patch quand le train monte et que le test reste plat (AFFIRMATION, 0.93)
- Hillclimbing —réduit→ coût par ticket à performance constante (AFFIRMATION, 0.9)
- Claude Opus —mesure→ 74,4 % de précision à 4,6 centimes par ticket (Opus 4.8, effort élevé, départ) (MESURE, 0.93)
- Claude Sonnet —mesure→ 90,5 % contre 78,6 % sur 14 tickets tenus à l'écart, pour environ un cinquième du coût (MESURE, 0.93)
- Claude Opus —améliore→ coût des tokens : 20 % de moins en entrée et sortie, 60 % de moins en lecture de cache (Opus 5.5 face à 4.8) (MESURE, 0.9)
- claude-api —mesure→ 66 % de réussite au départ, environ 88 % après hillclimbing (MESURE, 0.93)
- claude-api —observé_dans→ Claude écrit d'anciennes formes d'API issues de ses acquis malgré le contenu présent (AFFIRMATION, 0.9)
- Étape de réflexion sur les échecs —permet→ de détecter des tâches ou graders défectueux (AFFIRMATION, 0.9)

---
Canonical: https://www.thekb.eu/en/fiches/lancemartin-anthropic-automating-eval-design-hillclimbing-2026-09-28/