Skip to content

root / tags / terminal-bench-2-1

#Terminal-Bench 2.1

4 fiches

Architecture & Construction Auto-verified translation

Buzz (buzz.xyz) — Rapport de recherche pour présentation

Internal research report dated **August 12, 2026** consolidating, for presentation purposes, everything publicly documented about **Buzz** — **Block**'s humans + agents workspace, launched on **July 21, 2026** under the **Apache 2.0** license. It aggregates the two engineering posts already filed alongside the corporate announcement, the GitHub repository, press coverage, X, and **three independent hands-on accounts** that constitute the dossier's only non-self-reported data. **(A) A vocabulary gap documented by quotation**: **Jack Dorsey**'s launch tweet announces *"model-agnostic, decentralized, self-sovereign, and open source"*; Block's `ARCHITECTURE.md` states *"The relay is the single source of truth. All reads and writes flow through it. There is no peer-to-peer event exchange, no gossip, no replication."* The relay is therefore single and authoritative per community: Buzz's "decentralization" is an **organizational sovereignty** — self-hosting and portable identity — not network redundancy. **TFTC**'s formulation: *"Two of those three hold cleanly. The third needs a qualifier."* **(B) An asymmetry between demonstrated rigor and exploitation risk.** On one side, a rare degree of formalism for a v0.4.x/0.5.x: multi-tenant isolation specification **mechanized in TLA+**, authorization properties verified in **Tamarin**, a model-checked Git storage protocol, a hash-chained append-only audit log, 127 *event kinds*, NIP-01/42/98/34. On the other, channel membership is the unit of permission — *"channel membership is not fine-grained tool authorization"* (João Queirós) —, agents run in `--dangerously-skip-permissions` outside any sandbox on a human's machine, and observability is lacking: *"Buzz tells me an agent got a message. It doesn't tell me what happens next"* (DevTools Daily, which reports silent OOM kills). Block acknowledges it: *"the agent can do anything, and security rests entirely on restricting who can tell it what to do"*. **(C) The technical stack**, absent from the filed posts: **Rust** relay (Axum WS + REST), **Postgres**, **Redis**, **S3/MinIO** via Blossom, **Tauri + React** desktop client. Agent integration goes through **`buzz-acp`**, an **ACP** harness that plugs in goose, Codex and Claude Code and translates **ACP ↔ MCP**, plus **`buzz-agent`**, an in-house agent. The report corrects itself on one point: the *"+33% more work"* in Block's TL;DR is the **ratio of completed tasks (20 versus 15 out of 44)**, not a score gain — the score itself rises from 59.1% to 71.5%, i.e. **+12.4 points**.

#Buzz#buzz.xyz#Block

**Deep Research Veille Interne** — rapport non signé · produit le **12 août 2026** en préparation d'une présentation. Aucune URL publique ; source archivée dans `raw-data/`.

AI Coding Agents & Skills Auto-verified translation

Efficient Tokens & Effective Teams in Buzz

A **Block Engineering** benchmark post from **August 6, 2026**, signed by **Atish Patel**, about **Buzz** — the human + agent workspace launched on July 21 — asking a cost question: which agent team is **the cheapest one that reliably succeeds**? Three findings. **(A) A negative result, published in full**: on **Terminal-Bench 2.1**, **twelve team compositions** (pairs, triads, cheap swarms under a *frontier* model) were pitted against the solo agent each was built around, and **none beat it at equal cost**. The explanation is structural — a task that finishes in minutes *"doesn't have enough structure to divide"*, and *"More agents mostly buys you the cost of explaining it twice"*. **(B) The horizon reverses the result**: on **Long-Horizon Terminal-Bench** (44 tasks, one task worth hours of work, same lead **GPT-5.6 Sol** at *high* effort), solo finishes 15 tasks for 59.1%, +2 QuickBees 19 for 64.1%, +1 QuickBee +1 WorkerBee 19 for 69.5%, **+2 WorkerBees 20 for 71.5%** — a **+12.4-point** gain, of which 11.4 comes from tasks carried to completion. *"Same seats, opposite result, because the work is a different shape."* These runs ran at **3× the timeout**, solo included. **(C) Beyond a threshold, price stops buying quality**: solo on Terminal-Bench 2.1, **Opus 5 at *xhigh* effort is the most expensive run ($140.63) for 75.0%**, trailing six runs ranging from $20.08 to $109.82 and 79.5% to 88.4% — the stated cause is over-reasoning that drove 17 of 88 tasks to timeout. Among the six best runs, **a 5.5× price gap for an 8.9-point score gap**: *"choosing between them is not a quality decision at all. It is a budget decision."* The post proposes a taxonomy it owns as *ad hoc* — **QuickBee**, **WorkerBee**, **SmartBee**, plus the human as *"honorary bee"* — and two team forms, the permanent **Hive** that remembers your preferences and the disposable **Swarm** that remembers the project. Conditions: everything runs on **Harbor**, against real Buzz agents on a **live** relay, **one attempt per task, no retry**, prices fixed as of **2026-07-30**.

#Buzz#Block#agent teams

- **Atish Patel** — *« Building AI solutions @ Block »* · auteur unique du billet · publié le **6 août 2026** sur `engineering.block.xyz`.

AI Coding Agents & Skills Auto-verified translation

Introducing Muse Code and Muse Spark 1.2

Announcement from **Meta AI Research** published on **August 5, 2026** (stated reading time: 4 minutes, no individual byline): **Muse Code** in beta, *« a terminal coding agent »*, and the model that powers it, **Muse Spark 1.2**. Meta itself frames the launch: *« This marks our next step toward the frontier, with larger and much more capable models on the way. »* **Three architectural elements on the harness side.** **Asynchronous background agents** that *« remain active throughout each session, rather than being spawned for individual tasks »*, avoiding redundant information gathering and reducing the need for steering. A **local event log** where *« every model call, tool run, approval, and edit is appended »*, making the runtime a system that is *« replay-exact and restart-safe »*, able to resume exactly where it left off after a crash. And **three skills shipped out of the box**: `/plan` (turns a task into a plan submitted for approval), **`/grill`** (stress-tests the plan *« until it holds up »*), and `/goal`. **On the model side**, Meta claims **model-harness co-training** (*« to maximize harness compatibility »*, with harness trajectories sampled via rejection sampling and recipe optimizations for goals, compaction, and sub-agents), **long-horizon** training (whole-repo generation, end-to-end projects, self-research, with planning, goal conditioning, and context compaction), and a **self-improvement loop** where Muse Spark 1.1 generates the environments and instruction templates and then grades candidate solutions, producing a training set for the 1.2. **What the published charts show**, without the text commenting on it: the four comparisons — Terminal-Bench 2.1, DeepSWE 1.1, an internal Meta benchmark, and the GPU kernel optimization case study — place **Muse Spark 1.2 behind Opus 5 in all four cases**, including on Meta's own proprietary benchmark (70.6% versus 79.4%) and on the case study, where the model finishes fourth out of six (+68.7% versus +74.0%). **A reading caution on the version gain**: on the two public benchmarks, 1.1 is measured with `mini-swe-agent` and 1.2 with Muse Code, so the 6.7-point gap conflates model and harness. On the internal benchmark, the only comparison where no harness is mentioned, the 1.1 → 1.2 gap drops to **2.3 points**.

#Meta AI Research#Muse Code#Muse Spark 1.2

**Meta AI Research** — publication institutionnelle sans auteur nommé · sur `research.meta.ai`. Le billet renvoie à un **rapport** pour la méthodologie d'évaluation · non repris ici.

Economy & Market Auto-verified translation

GPT-5.6 Sol, Terra, Luna : comment OpenAI rebat les cartes du coding agentique et du pricing

SFEIR analysis (firm's voice) of the general availability, on July 9, 2026, of **GPT-5.6** by OpenAI — not a single model but a **family of three tiers**: **Sol** (long-horizon/cyber/science flagship, the only one to unlock the "max" and "ultra" modes), **Terra** (everyday balanced tier, ~half the price of GPT-5.5), and **Luna** (fast/economical, high volume). All three share ~**1.05M tokens** of context, **128k** output tokens, and a knowledge cutoff of **February 16, 2026**. The most structuring fact is not a score but an **aggressive pricing grid** (Sol $5/$30, Terra $2.50/$15, Luna $1/$6 per million tokens): Sol keeps the previous flagship's price while being more capable, forcing the comparison onto the **capability-to-cost ratio**. Two billing subtleties (cache writes billed at **1.25×**, a surcharge beyond **272k** tokens) make the grid misleading until one has measured how much context the agent re-reads (read/write ratio ~**153:1** in agentic coding). Engineer's verdict, claimed to be neutral (SFEIR is both a **Google Cloud Premier** partner *and* an **Anthropic** partner): **no one sweeps every table** — GPT-5.6 dominates Terminal-Bench 2.1 and the Coding Agent Index (at a third of the cost per task), Claude stays ahead on SWE-Bench Pro (~15 pts); METR flagged a record **reward hacking** rate on Sol. Conclusion: "stop looking for the champion, learn to route" — the model is a commodity, the durable advantage lies in **Context/Harness Engineering**.

#GPT-5.6#Sol#Terra

SFEIR (voix éditoriale du cabinet)