Scaling CI Test Selection for Agentic Coding at Anthropic

Key takeaways

1. The causal chain, not just the number. 8× more code shipped per engineer per quarter, 80% of it written by Claude, plus a significant role for AI in PR review and approval; ×10 on the number of tests with nearly constant headcount. Result: ×25 on CI jobs in six months. « Writing code is no longer the constraint, and once PR review gets accelerated, CI starts feeling the pressure. » The bottleneck doesn't disappear, it moves down a level — an infrastructural variant of the "verification bottleneck" from [[sfeir-code-review-anneau-contraintes-2026-07-30]].
2. The argument specific to agents. , the most transferable point in the text. Running everything on every change works up to a point, then gates become long, costly, and unreliable — and therefore ignored. Above all: a human knows how to dismiss a test failure that doesn't concern them, an agent doesn't — it needs more context and direction. Given a valid and relevant set of tests, it self-verifies and iterates on its own. Test selection stops being a cost optimization and becomes a condition for autonomy.
3. The v0 architecture and its original sin. keeping a per-test history required a single writer, hence a single process, hence no horizontal sharding possible. State lived inside the process.
4. The physics of lag. 20 minutes of lag = tens of thousands of updates left unapplied. Three distinct consequences — a bad merge fails a test for everyone and triggers unnecessary investigations; a dependency that starts flaking produces reds that block merges; a fixed or added test doesn't run until the listener has caught up, risking a regression.
5. The three patches and their half-life. (1) doubling the cores — 70 days; (2) sharding by package — the listener didn't need a single global writer, but a single writer per package, and Claude generated the code — 29 days; (3) daily restarts — less than a day, with only four bugs found, a memory allocator change with no effect, and the deliberate refusal to profile the memory of a singleton already under load.
6. Delegated supervision. , a detail worth keeping: a long-running session in an internal version of Claude Tag dedicated to the service, which alerts the author as soon as the listener falls more than 50,000 jobs behind and picks the conversation back up where it left off. For months, without having to reload the context of past attempts. And « Claude often argued for an overhaul, but we usually settled on another patch ».
7. The overhaul. giving the service a foundation — a magasin de données en mémoire. Any listener worker processes any result, appends it to a log, and moves on without keeping anything: stateless, hence horizontally scalable. A separate consumer aggregates the log into per-test history every few seconds. More expensive to run, but much easier to scale and profile than a fragile singleton. The fine-tuning (log size, number of workers) was done largely by Claude on its own.
8. The economics of the decision have changed. Three weeks for one engineer, versus about a quarter a year earlier. That's the real shift: half-measures buy less and less time while a full overhaul costs less and less — « much more sustainable now that writing code is no longer the bottleneck ».
9. The four rules he draws from it. assume the architecture will face ×25 the load within two quarters, whether built or bought; accept that over-engineering recedes as a mistake — aim for 10-20× the perceived scale as early as v0 if the budget allows; instrument services so they serve as Claude's eyes and ears, at minimum the invariant "as many CI jobs in as out"; take state out of the process from the start, never run a critical service as an unmeasurable single instance.
10. The shape of the load changes, not just its volume. Claude favors smaller, more granular PRs (one more argument against running every test on every PR), the activity floor rises because agents push at night and on weekends, but the load stays bursty since humans still drive and approve a significant share of PRs.
11. The final bet. , to be checked in twelve months: « I anticipate horizontally scaled test selection architecture will become industry standard » as agents produce both more PRs and more tests.