Pacing the AI Frontier: METR Evaluators, SALT-Style Cap

Key takeaways

1. What "pacing" means, and what it doesn't. neither halting training nor freezing technical progress, but the time to align and secure models — and to have a third party verify it. Progress "will still seem fast."
2. The 2023 question, finally answered. Amodei then judged a pause absurd, for lack of knowing what to do with the extra time: studying the alignment of those models was like "studying human psychology by doing experiments on bacteria." Current models are, by contrast, a mine of lessons about what goes wrong — one or two more years before critical thresholds would therefore genuinely reduce risk.
3. The four uses of the time gained. operational excellence, alignment, interpretability, testing and evaluation — what's missing often being not a theory but execution. Hence a rare admission from this pen: the alignment incidents Anthropic recently reported stem partly from imperfect filtering of broken RL environments, carried out with its providers "reasonably diligently, but not well enough." The intended reference point is commercial aviation — millions of incident-free flights, but it took time.
4. Why evaluation gets harder as capability grows. a smarter model is better able to deceive tests, and so to appear aligned while concealing a serious problem. Interpretability serves as a cross-check — it was used to examine unverbalized motivations during recent incidents — but its methods don't always yield a reliable result.
5. The quantified bet on the swarm. within 6-12 months, a swarm as misaligned but more capable than the one in [[mollick-agency-and-agents-twilight-factory-2026-08-31]] could take hold of the internet with a persistent botnet, for hundreds of billions of dollars. Refusing to attribute this to a single actor is the core of the argument: similar, less serious incidents have occurred elsewhere, Anthropic included.
6. The detail of the embedded-evaluator contract. , which separates the symbol from the mechanism: on-site offices, badges, company laptops, permissions comparable to internal risk-evaluation teams, live conversations with employees. And above all the right to publish without editorial control from Anthropic — redactions limited to safety, legal privilege, commercial sensitivity, and third-party confidentiality; "we can't redact findings just because they are unfavorable," and evaluators can publicly state that a redaction removed something important.
7. The three claimed benefits. verifiability (a neutral third party sees the detail, where any pacing commitment carries ambiguity and the letter-versus-spirit problem), transparency (it is still Anthropic that chooses what its model cards omit), and a second opinion free of commercial incentive.
8. The two paths within democracies. targeted regulation (transparency and third-party audit), which also covers holdouts but takes time; and, in parallel, voluntary standards among companies — for which the state need not participate but must issue a narrow antitrust exemption. The mechanism suggested by Demis Hassabis is cited as a possible path.
9. Two ways to set the pace, and their asymmetry. by observed capabilities — checkpoints of the type "if the model can evade common sandboxing methods, then it requires such-and-such alignment certifications" — or by inputs (training compute, the nature of the runs, internal use of AI to improve AI). Amodei prefers the former: inputs are more gameable than external behavior.
10. The geopolitical constraint, stated explicitly. one can only slow down by the margin of the lead over CCP-linked projects. Three measures to defend it — chips and manufacturing equipment (including smuggling and remote access to datacenters outside China), unauthorized distillation, weight theft — with the aim of widening the lead over 3-5 years. All three strands are documented on the threat side in [[anthropic-threat-intelligence-misuse-report-2026-09-10]], published a few days earlier. The counterintuitive argument made explicitly: these measures make a future agreement more likely, because they increase democracies' leverage.
11. The four levels of global coordination. , in increasing order of difficulty: (1) banning clearly dangerous uses, such as biological weapons — "probably possible"; (2) mutually testing models before release via a standards body — creatable, but hard to give real teeth and hard to verify against secret models; (3) a speed cap on recursive self-improvement, analogous to the SALT treaties — "at the edge of the possible"; (4) a general pause, unlikely in the short term. Cross-cutting rule: any agreement must have bulletproof verifiability, or stay limited enough that a defection isn't militarily existential.
12. The corpus counterpoint. On August 10, 2026, Zuckerberg treated recursive self-improvement as an unavoidable competitive dilemma, with the safeguard that a "significant majority" of compute remain human-directed — no numerical threshold, no designated verifier, no sanction. Amodei takes the problem from the other end: the verifier first, thresholds second, since without an embedded third party no pacing commitment is verifiable.