# malhotra-anthropic-ci-test-impact-analysis-2026-09-14

## Veille

Engineering post from **Anthropic** published on **September 14, 2026** on claude.com/blog (~1,400 words, five sections), authored by **Sachin Malhotra**. It recounts scaling the **test impact analysis** service — selecting which tests to run per change — under the pressure of agentic coding. The input numbers and their chain: engineers ship on average **8×** more code per quarter than in 2021-2025, **Claude writes 80% of it** and also carries significant weight in PR review and approval; the number of tests was multiplied by **10** with nearly constant headcount; hence **25× CI jobs in six months**. **(A)** The v0 architecture rests on two components that must stay synchronized — a *listener* that records the results of each run, a *selector* that decides which tests run on each PR — but as a **single process**, a per-test history assuming a single writer. **(B)** The failure mode is the listener falling behind the PR queue: *« 20 minutes of listener lag can translate into tens of thousands of test updates not being applied »*, with three named consequences — wasted investigations, *flaky reds* that block merges, a fixed or added test that doesn't run. **(C)** Three patches whose lifespan collapses — **70 days, 29 days, less than a day**: a bigger machine, sharding by package, daily restarts. Then the overhaul: an **in-memory** magasin de données en mémoire, **stateless** *listener* workers that are therefore horizontally scalable, a log aggregated every few seconds. **Three weeks for one engineer; a quarter, a year earlier.** Extends the 80% of [[clinton-anthropic-secure-ai-native-sdlc-2026-07-21]] through infrastructure.

## Titre Article

Agentic coding is straining CI. Here's how we scaled test impact analysis at Anthropic

## Date

2026-09-14

## URL

https://claude.com/blog/agentic-coding-is-straining-ci-heres-how-we-scaled-test-impact-analysis-at-anthropic

## Keywords

continuous integration, CI, test impact analysis, test selection, listener, selector, listener lag, 25× CI jobs in six months, 80% of code written by Claude, flaky reds, stale selection data, single process, single writer, sharding by package, horizontal scalability, stateless workers, magasin de données en mémoire, log, memory saturation, service overhaul, planning for the exponential, over-engineering, v0 sized 10-20×, instrumentation, Claude's eyes and ears, Claude Tag, long-running session, smaller PRs, bursty load

## Authors

Sachin Malhotra (ingénieur, Anthropic), sur le blog claude.com.

## Ton

**Profile**: first-person engineering account, framed as a deliberate post-mortem. Audience: platform, CI, and developer-tooling teams who will see the same curve coming. Direct English, backed by an architecture diagram and a before/after chart.

**Style**: the text is built on **a decreasing sequence of numbers** — 70 days, 29 days, less than a day — and this sequence *is* the argument, far more than the techniques used. The author himself downgrades them upfront: buying a bigger machine, parallelizing, restarting the service *« (yeah, this one still works surprisingly well) »* are *« common and not the insights to take from this article »*. The lesson shifts from the means to the **pace**: each technique buys a fraction of the time it bought a year ago, while the full overhaul also costs a fraction of what it used to cost. Two conversation screenshots illustrate the account, honestly labeled — one *« recreated, based on real events »*, the other verbatim with redactions.

**Epistemic position**: entirely *first-party* and unauditable — the company measures its own tooling with its own counters, and the most-cited figures (25×, 8×, 80%) are self-reported. Two admissions give the text its value: the trend was readable but *« ownership was murky — no one wanted to own another piece of infrastructure »*; and Claude, hooked up to continuous monitoring, **had been arguing for the overhaul for months** while the team *« settled on another patch »*. The author also bounds the scope of the incident rather than dramatizing it: no untested code shipped to production, selection was only working off stale data.

## Pense-betes

- **The causal chain, not just the number.** 8× more code shipped per engineer per quarter, **80% of it written by Claude**, plus a significant role for AI in PR review and approval; **×10 on the number of tests** with nearly constant headcount. Result: **×25 on CI jobs in six months**. *« Writing code is no longer the constraint, and once PR review gets accelerated, CI starts feeling the pressure. »* The bottleneck doesn't disappear, it moves down a level — an infrastructural variant of the "verification bottleneck" from [[sfeir-code-review-anneau-contraintes-2026-07-30]].
- **The argument specific to agents**, the most transferable point in the text. Running everything on every change works up to a point, then gates become long, costly, and **unreliable** — and therefore ignored. Above all: a human knows how to dismiss a test failure that doesn't concern them, an agent doesn't — it needs more context and direction. Given a **valid and relevant** set of tests, it self-verifies and iterates on its own. Test selection stops being a cost optimization and becomes a **condition for autonomy**.
- **The v0 architecture and its original sin**: keeping a **per-test** history required a single writer, hence a single process, hence **no horizontal sharding possible**. State lived inside the process.
- **The physics of lag**: 20 minutes of lag = tens of thousands of updates left unapplied. Three distinct consequences — a bad merge fails a test for everyone and triggers unnecessary investigations; a dependency that starts flaking produces *reds* that block merges; a fixed or added test doesn't run until the listener has caught up, risking a regression.
- **The three patches and their half-life**: (1) doubling the cores — **70 days**; (2) **sharding by package** — the listener didn't need a single global writer, but a single writer *per package*, and Claude generated the code — **29 days**; (3) daily restarts — **less than a day**, with only four bugs found, a memory allocator change with no effect, and the deliberate refusal to profile the memory of a singleton already under load.
- **Delegated supervision**, a detail worth keeping: a **long-running session in an internal version of Claude Tag** dedicated to the service, which alerts the author as soon as the listener falls more than **50,000 jobs behind** and picks the conversation back up where it left off. For months, without having to reload the context of past attempts. And *« Claude often argued for an overhaul, but we usually settled on another patch »*.
- **The overhaul**: giving the service a foundation — a magasin de données **en mémoire**. Any *listener* worker processes any result, appends it to a **log**, and moves on without keeping anything: stateless, hence horizontally scalable. A separate consumer aggregates the log into per-test history every few seconds. More expensive to run, but **much easier to scale and profile** than a fragile singleton. The fine-tuning (log size, number of workers) was done largely by Claude on its own.
- **The economics of the decision have changed.** Three weeks for one engineer, versus about a quarter a year earlier. That's the real shift: half-measures buy less and less time while a full overhaul costs less and less — *« much more sustainable now that writing code is no longer the bottleneck »*.
- **The four rules he draws from it**: assume the architecture will face **×25 the load within two quarters**, whether built or bought; accept that over-engineering recedes as a mistake — aim for **10-20×** the perceived scale as early as v0 if the budget allows; **instrument services so they serve as Claude's eyes and ears**, at minimum the invariant "as many CI jobs in as out"; take state out of the process from the start, never run a critical service as an unmeasurable single instance.
- **The shape of the load changes, not just its volume**: Claude favors **smaller, more granular** PRs (one more argument against running every test on every PR), the activity floor rises because agents push at night and on weekends, but the load **stays bursty** since humans still drive and approve a significant share of PRs.
- **The final bet**, to be checked in twelve months: *« I anticipate horizontally scaled test selection architecture will become industry standard »* as agents produce both more PRs and more tests.

## RésuméDe400mots

Sachin Malhotra, an engineer at Anthropic, recounts how the company had to overhaul its test impact analysis service under the pressure of agentic coding. The starting point is a chain of figures: engineers ship on average eight times more code per quarter than between 2021 and 2025, Claude writes 80% of it and plays a major part in PR review and approval, the number of tests was multiplied by ten with nearly constant headcount. The product of all this is a twenty-five-fold increase in CI jobs in six months. Writing code is no longer the constraint; once review is accelerated, it is CI that absorbs the pressure.

The service rests on two deterministic components that must stay synchronized: a listener, which records the results of each run, and a selector, which reads this history to decide which tests run on each PR. Keeping a per-test history required a single writer, hence a single process, hence the impossibility of sharding horizontally. The failure mode is the listener falling behind: twenty minutes is enough to leave tens of thousands of updates unapplied, which triggers unnecessary investigations, lets flaky tests block merges, and prevents a fixed or added test from running.

Three fixes follow one another, each with a collapsing lifespan: doubling the cores holds for seventy days; sharding by package — the listener didn't need a single global writer but one per package, and Claude wrote the code for it — holds for twenty-nine days; daily restarts hold for less than a day. In the meantime, a long-running session in an internal version of Claude Tag monitors the service and alerts beyond fifty thousand backlog jobs; it argues for the overhaul for months, but the team prefers the patch.

The overhaul gives the service a magasin de données en mémoire. Each listener worker processes any result, appends it to a log, and moves on without retaining anything: stateless, hence horizontally scalable. A separate consumer aggregates the log into per-test history every few seconds. More expensive to run, but scalable and profilable. Three weeks for one engineer, versus about a quarter a year earlier.

The lessons come down to four rules: anticipate the exponential and assume a load twenty-five times higher within two quarters, size the v0 at ten to twenty times the perceived scale, instrument services so they serve as Claude's eyes and ears, and keep state out of the process from the start.

## GrapheDeConnaissance

- Sachin Malhotra —travaille_chez→ Anthropic (ORGANISATION, 0.95)
- Anthropic —mesure→ 8× plus de code livré par ingénieur et par trimestre qu'en 2021-2025 (MESURE, 0.93)
- Anthropic —mesure→ Claude écrit 80 % du code livré et pèse largement dans la revue des PR (MESURE, 0.93)
- Anthropic —mesure→ ×10 sur le nombre de tests et ×25 sur les jobs CI en six mois (MESURE, 0.94)
- IA agentique —observé_dans→ déplacement du goulot de la génération de code vers l'intégration continue (AFFIRMATION, 0.92)
- Test impact analysis —s_applique_à→ Intégration continue (METHODOLOGIE, 0.94)
- Test impact analysis —utilise→ historique de résultats par test et pertinence par package (CONCEPT, 0.92)
- Test impact analysis —permet→ à un agent de s'auto-vérifier et d'itérer sur un jeu de tests pertinents (AFFIRMATION, 0.9)
- retard de listener —réduit→ fraîcheur des données de sélection des tests (CONCEPT, 0.9)
- retard de listener —mesure→ 20 minutes de retard = des dizaines de milliers de mises à jour non appliquées (MESURE, 0.92)
- retard de listener —observé_dans→ investigations inutiles, flaky reds bloquant les merges, test corrigé qui ne tourne pas (AFFIRMATION, 0.91)
- Claude Tag —observé_dans→ session longue de supervision du service, alerte au-delà de 50 000 jobs de retard (AFFIRMATION, 0.9)
- Claude Tag —recommande→ refondre le service plutôt qu'enchaîner les rustines (AFFIRMATION, 0.88)
- Magasin de données en mémoire —permet→ des workers listener sans état, donc scalables horizontalement (AFFIRMATION, 0.93)
- scalabilité horizontale —résout→ saturation mémoire du processus unique (CONCEPT, 0.91)
- Anthropic —mesure→ refonte livrée en trois semaines par un ingénieur, contre environ un trimestre un an plus tôt (MESURE, 0.92)
- Sachin Malhotra —affirme_que→ "Always plan for the exponential" (CITATION, 0.95)
- Sachin Malhotra —recommande→ supposer une charge ×25 dans les deux trimestres, qu'on construise ou qu'on achète (AFFIRMATION, 0.94)
- Sachin Malhotra —recommande→ dimensionner la v0 pour 10-20× l'échelle perçue si le budget le permet (AFFIRMATION, 0.92)
- Sachin Malhotra —recommande→ instrumenter les services pour qu'ils servent d'yeux et d'oreilles à Claude (AFFIRMATION, 0.92)
- Sachin Malhotra —recommande→ sortir l'état du processus dès le départ, jamais de service critique en instance unique (AFFIRMATION, 0.94)
- Sachin Malhotra —prédit→ l'architecture de sélection de tests scalée horizontalement deviendra un standard de l'industrie (AFFIRMATION, 0.88)
- IA agentique —observé_dans→ PR plus petites et granulaires, activité nocturne et de week-end, charge restant en rafales (AFFIRMATION, 0.9)

---
Canonical: https://www.thekb.eu/en/fiches/malhotra-anthropic-ci-test-impact-analysis-2026-09-14/