Measurements

Part V is numbers instead of impressions: what agent work costs, how benchmarks are scored, and how often the tools change. Each chapter says how its figures were collected, ships the dataset behind them, and is plain about what it did not measure.

Begin with SWE-bench explained, which shows how a resolve rate is produced, what the Verified subset changed, and what five papers found about harness effects, lucky passes, contamination and realistic prompts. SWE-bench Verified task by task then reads the leaderboard's own per-task records for 11 models on one harness: how task size predicted failure, what a cap on model calls keeps and saves, and what a resolved task cost. Flaky tests works out what a green rerun tells an agent; a test that fails 1 percent of the time needs 299 runs to show itself. Release cadence counts third-quarter 2026 releases from npm publish times: 79 for Claude Code, 38 for Codex CLI and 15 for Gemini CLI.

What one coding task costs is a method that leaves the measuring to you: a calculator that reproduces the vendors' own worked examples and a dataset of published numbers. Time-to-green began the same way and now has its first results: ten small tasks run three times each with two Claude Code models on 30 October 2026, all 60 attempts green, with a row per attempt, the token ledger and redacted transcripts published beside it. The release-cadence chapter also counts how many releases have shipped since a chapter's version stamp, which is the quickest way to judge how far one has aged.