# osmani-agentic-verification-2026-10-14

## Veille

Post by **Addy Osmani** on his Substack *Elevate*, published on **October 14, 2026**; ~1,900 words. Thesis: with current models, the bottleneck is no longer writing code but **verifying** its correctness; human work shifts from *"review every line"* to *"build loops the agent can run itself."*

## Titre Article

Agentic Verification — The bottleneck has moved from writing code to knowing it's correct.

## Date

2026-10-14

## URL

https://addyo.substack.com/p/agentic-verification

## Keywords

agentic verification, feedback loop, self-verifying agent, end-to-end tests, E2E, property-based tests, property-based testing, differential tests, oracle, migration, invariants, idempotence, determinism, flaky tests, flaky tests, feature map, formal methods, TLA+, Lean, success criterion, evidence review

## Authors

Addy Osmani — ingénieur (Anthropic, cf. fiches antérieures) ; billet sur son Substack *Elevate*.

## Ton

Profile: a practice guide aimed at engineers working with coding agents, in a confident, didactic tone. The piece opens with a thesis (verification is the bottleneck), redefines the role of tests, then walks through three families of checks, each with its benefits, limits, and starting advice, before a "what good looks like" list and a closing directive. "Note" boxes point to adjacent practices (feature maps, formal methods, reviewer agents). The argument rests on the author's own experience and unnamed teams' feedback, without figures or studies. Comments extend the discussion: IronBee (also verify the API behind the UI) and a reader who suggests keeping acceptance criteria hidden from the building agent.

## Pense-betes

- **Thesis**: the bottleneck shifts from writing to verification; the human defines success and reviews evidence, not diffs.
- **Loop**: write → check → read → repeat, before any human look; a much better loop beats a slightly better model.
- **Capitalizing taste**: every agent error becomes shared config, a hook, or a test.
- **E2E**: an oracle faithful to the user; must say *what* failed (screenshots, traces); few and high-value, since they are slow and flaky.
- **Properties**: *"Example tests say what should happen; properties say what must not"*; start on a pure function, write the property in plain English first, keep the minimized case as a regression.
- **Differential**: old system = oracle; random inputs or replayed real traffic; normalize the insignificant; a justified list of approved changes.
- **Healthy loop**: sub-second for the core loop, seeded randomness, frozen clock, controlled network; flaky tests pulled from the suite; short output; E2E and diff tests before integration.
- **Adjacent habits**: restate success criteria before coding, a reviewer agent in a fresh context, inspect generated tests.
- Related to [[monperrus-end-of-code-review-agents-supersede-2026-06-11]] and [[osmani-three-joys-engineering-agents-2026-10-09]].

## RésuméDe400mots

Addy Osmani starts from an observation: giving an agent a way to verify its own work strongly improves the quality of what it produces. What made "good code" expensive — tests, benchmarks, specifications, real end-to-end paths — can now be delegated to the agent. Human work therefore shifts from line-by-line review to building loops the agent runs on its own, and taste gets capitalized: every error becomes a shared configuration, a hook, or a test. A human remains responsible for what ships, but reviews evidence rather than diffs.

In his view, models are now good enough that verification is the bottleneck. An agent that cannot check itself guesses, and correction then requires a slow, costly, unscalable human round trip. An agent that tests itself writes, checks, reads the result, and tries again before any human steps in. Tests change role: they become how the agent perceives the world and knows it is done.

End-to-end tests are the oracle of choice: they describe what the user does and are hard to fool, unlike a mock-based unit test, which can share the agent's own misunderstanding. They must point to the failing step, turning a vague instruction into a target. Slow and flaky, they should stay few in number.

Property-based tests state what must never happen; tools such as fast-check, Hypothesis, or proptest generate thousands of inputs and shrink failures down to a minimal case. Common properties: round-trips, invariants, idempotence, equivalence with a simple version. They stop the agent from picking examples that merely confirm its own logic and serve as a specification.

For a rewrite, differential tests turn the old system into the oracle: identical inputs, random or drawn from real traffic, output comparison, normalizing away the insignificant, and a justified list of intended changes, until the old system can be retired.

The loop must be fast and deterministic: an agent facing slow or flaky tests learns to retry rather than fix, or spaces out its runs and loses track of causes. Osmani recommends a core loop under a second, controlled randomness, immediate exclusion of flaky tests, concise output, and heavier tests before integration. His conclusion: write the check before the code.

## GrapheDeConnaissance

- Addy Osmani —publie→ article Agentic Verification (DOCUMENT, 0.98)
- Addy Osmani —affirme_que→ le goulot des agents de codage est la vérification, non l'écriture du code (AFFIRMATION, 0.95)
- Addy Osmani —affirme_que→ une bien meilleure boucle de rétroaction vaut mieux qu'un modèle un peu meilleur (AFFIRMATION, 0.92)
- Agentic verification —améliore→ Agents de codage (TECHNOLOGIE, 0.9)
- Agentic verification —utilise→ Tests E2E (METHODOLOGIE, 0.93)
- Agentic verification —utilise→ Property-based testing (METHODOLOGIE, 0.93)
- Agentic verification —utilise→ Differential testing (METHODOLOGIE, 0.93)
- Property-based testing —utilise→ fast-check (TECHNOLOGIE, 0.9)
- Property-based testing —utilise→ Hypothesis (TECHNOLOGIE, 0.9)
- Property-based testing —utilise→ proptest (TECHNOLOGIE, 0.9)
- Differential testing —s_applique_à→ migration logicielle (CONCEPT, 0.9)
- Tests instables —réduit→ Agentic verification (METHODOLOGIE, 0.85)
- Addy Osmani —recommande→ écrire le contrôle avant le code (AFFIRMATION, 0.95)
- Addy Osmani —recommande→ une boucle de tests rapide et déterministe, sous la seconde pour la boucle cœur (AFFIRMATION, 0.9)
- Addy Osmani —référence→ Lauren (@poteto) (PERSONNE, 0.85)
- Lauren (@poteto) —recommande→ Feature map (CONCEPT, 0.85)
- Boris Cherny —utilise→ TLA+ (TECHNOLOGIE, 0.85)
- Boris Cherny —utilise→ Lean (TECHNOLOGIE, 0.85)
- Boris Cherny —a_créé→ Claude Code (TECHNOLOGIE, 0.85)

---
Canonical: https://www.thekb.eu/en/fiches/osmani-agentic-verification-2026-10-14/