Build-Eval and Hillclimb: Claude Code's claude-api Skill

Key takeaways

1. Four marks of a good evaluation. Tasks drawn from production; a score that rises with a stronger model and more effort; best model clearly below 100%; low variance. Warning sign: a task that fails on every run regardless of how many times it's repeated is probably ambiguous or poorly graded.
2. Criterion for a good task. Two domain experts reach the same verdict, and everything the grader checks is stated in the task. Variance can also come from configuration (effort misapplied) or environment (residual state, git history that hands over the answer).
3. Adversarial sampling. Choosing the cases the current model fails amounts to sampling the valleys of a single model. The cases kept are those judged difficult by a human, and those drawn from bugs or tickets. Raw user traffic can skew toward the easy, since users tend to attempt what they expect to succeed at.
4. build-eval, source order. Production transcripts (after a question on retention and sensitive data), bug reports and tickets, five to ten hand-written cases, cases synthesized from the code. A generated page shows all inputs and waits for the user's confirmation.
5. Grader choice. The cheapest one that fits: programmatic checking if outputs are constrained; otherwise LLM-as-judge with a checklist of verifiable claims (not a 1-to-5 scale), a judge model distinct from the model being tested, and a blind comparison in random order against a reference.
6. Diagnostic checks on the baseline. Grader replayed twice on the same output; plumbing (timeouts, API errors, truncated responses); headroom, with a warning if the baseline exceeds about 95%, in which case targeting cost or latency instead.
7. Where to apply hillclimbing. A surface that is cheap to change (prompts, skills rather than harness code), a score attributable to that surface (a skill's trigger rate and its description), a bounded objective. Cost is a robust target even on a saturated evaluation.
8. Three safeguards against overfitting. Separate train and test; never paste the content of failures into the prompt; keep answers structurally out of the model's reach. Examples of leakage: an OCR tool added because the evaluation happens to benefit from it, or a public repository from which the harness retrieves the solution.
9. hillclimb loop. Choice of the modifiable surface (system prompt, skills, tool descriptions, model and effort, harness code), an objective, a random train/test split, a prior check that noise is lower than the smallest useful gain. Each round reads the train transcripts and proposes a patch; kept if both train and test rise, reverted otherwise. Delivers the version that scores best on test, with confidence intervals; if the gain is within the noise, the recommendation is not to merge.
10. Reflection step. When the score stalls for two or three rounds, no editing happens: each remaining failure is classified by cause. This step revealed skill content that was present but ignored, Claude writing outdated API forms drawn from its training (a fixed thinking budget instead of adaptive reasoning), and flawed tasks or graders.
11. Cost example (customer support). 44 tickets, 30 for research and 14 held out for test. Starting point: Opus 4.8 at high effort, 74.4% accuracy, 4.6 cents per ticket. Prompt audit, then Opus 5.5 at low effort (87.8%, 1.9 cents), then Sonnet 5 at low effort (88.9%, 1 cent), finally an improved prompt (98.9%). On the 14 held-out tickets: 90.5% versus 78.6%, for about a fifth of the cost.
12. Performance example (claude-api skill). From 66% to 74% by adding eight missing features, 77% after fixing the C# and Java type tables, 80% with a mapping table from old to new API forms, about 88% after fixing two flawed tasks and graders.