Coding agent time-to-green: ten small tasks, 60 Claude Code runs

This page started as a protocol and now has its first results. You get ten small, fully specified coding tasks, a definition of "green" that a script can check, a runner that times an agent from prompt to first green and writes one CSV row per attempt, and one recorded run of the whole protocol: on 30 October 2026 we ran the ten tasks three times each with Claude Code 2.1.314 in headless mode, once with its default model and once with Sonnet, 60 attempts in all. Every attempt went green; the median time to green was 32.4 seconds for the default model and 25.6 seconds for Sonnet. The results below give the times, turns, tokens, estimated cost and refusals, then the acceptance tests we wrote, the method and the limits. The rows for every attempt, the token ledger and the redacted transcripts are published with the script that recomputes every number from them.

First results: 60 runs of Claude Code, 30 October 2026

We ran the protocol on 30 October 2026, between 18:42:57 and 19:12:46 UTC: all ten tasks, three attempts each, with two models, for 60 attempts. The system under test was Claude Code 2.1.314 in print mode (claude -p), once with its default model, which resolved to claude-opus-5-5, and once with --model sonnet, which resolved to claude-sonnet-5-5. Every attempt went green.

All 60 attempts together used 3,934,733 tokens: 670 uncached input, 507,746 written to the prompt cache, 3,271,691 read from it and 154,626 output. The default-model attempts used 2,325,738 of them and the Sonnet attempts 1,608,995. Cache reads are 83% of the total: every turn sends the conversation so far again, and the cache makes that cheaper, not free.

The dollar figures are an estimate, not a bill. They are Claude Code's own client-side figure (total_cost_usd), which reported a list-price basis on every attempt: $6.22 for all 60, $4.63 for the 30 default-model attempts and $1.59 for the 30 Sonnet attempts, or $0.154 and $0.053 per green attempt. The runs used a subscription login, not an API key, so this is what the same calls would be priced at through the API, not an amount anyone was charged.

Per task

Three attempts per cell is the minimum this page asks for, and the table shows why one is not enough: T03 on the default model took 23, 49 and 26 seconds for the same prompt on the same starting files.

What the numbers say, and what they don't

On tasks this small, pass rate does not separate the two models: both went 30 for 30. The differences are in time and cost. Sonnet had the lower median on 8 of the 10 tasks, and its estimated cost per green attempt was about a third of the default model's ($0.053 against $0.154).

No attempt came near a cap. The most turns in one attempt were 14 of the 30 allowed, the highest estimated cost was $0.216 of the $2.00 allowed, and the slowest attempt took 56.4 seconds against a 900-second timeout. So this run says nothing about the long tail of failed attempts that the Fail-Fast paper describes; that needs bigger tasks.

Claude Code refused 66 tool calls in 55 of the 60 attempts: 41 refusals across all 30 default-model attempts and 25 in 25 of the 30 Sonnet attempts. Those refusals are the containment working, not errors, and none stopped an attempt from going green. The commonest was running the acceptance test and the repo's own suite in one command, which an allow rule for the green command alone does not cover; others were inline Python scripts and shell loops. Partly as a result, most attempts ended without the agent having seen its acceptance test pass: it stopped when it judged the change done, and the green command that the bench ran straight after the agent exited is what found it green.

Method

• Tasks. The ten rows of the task file. The prompt was the task text exactly, and the agent was not told the green command.
• Acceptance tests. The first version of this page named ten test modules but did not ship them. We wrote them on 30 October, before any attempt, from the published task text alone, and they are now published: t01, t02, t03, t04, t05, t06, t07, t08, t09, t10. Where the task text leaves a choice open, the test accepts any reasonable answer: T01 may print numbers or numeric strings, and T10 may keep the report's DD/MM/YY timestamps or use ISO ones. Each module also runs a frozen copy of the shipped suite, so "text output unchanged" is checked against the original tests even if the agent edits the repo's own test file. T05's behaviour was already right, so its check is a mutation test: the repo's own tests must fail against two broken versions of head -n 0. T08 to T10 drive the script as a command line. Before the run we checked every module both ways: it fails on the starting code and passes after a hand-written fix, and a deliberately weak T05 test stays red.
• T06 and T07. As this page specifies, the one characterization test that pinned the old behaviour was replaced by the corrected expectation, in the repo's test file and in the acceptance test, so each task starts red on exactly one test. For T06:

    def test_first_and_last_are_correct_across_a_month_boundary(self):
        # Corrected expectation (time-to-green T06): 31 August is first, 1 September is last.
        out = run(["2026-08-31 23:59:59 INFO x\n", "2026-09-01 00:00:01 INFO y\n"])
        self.assertIn("first: 31/08/26 23:59:59\n", out)
        self.assertIn("last:  01/09/26 00:00:01\n", out)

For T07, test_level_match_is_case_sensitive became test_level_match_is_case_insensitive_and_printed_upper_case: three lines logged as error, ERROR and Error must print one ERROR row with a count of 3, and no lower-case or mixed-case row.

• Rebuilding the starting repos. time-to-green-make-task-repos.py builds all ten from the six tutorial files already on this site and the ten acceptance tests, applies the two swaps, and checks that they reproduce the published T06 and T07 tests byte for byte. We ran it on copies of those files in an empty folder, and its ten folders matched the ones the attempts started from, file for file.
• Isolation. Every attempt started in a fresh copy of its task's repo, outside any site folder. The command line, with the task text as the prompt (the exact argument list of each attempt is in the transcripts):

claude -p "<task text>" --output-format stream-json --verbose --max-turns 30 --max-budget-usd 2.00
    [--model sonnet] --restricted --tools Bash,Read,Edit,Write,Glob,Grep --strict-mcp-config --no-chrome
    --no-session-persistence --permission-mode dontAsk --permission-prompts none
    --allowedTools Read Glob Grep "Edit(./**)" "Write(./**)"
        "Bash(python -m unittest tasks.tNN_test)" "Bash(python -m unittest -v tasks.tNN_test)"
    --disallowedTools WebFetch WebSearch "Edit(./tasks/**)" "Write(./tasks/**)"

In dontAsk mode anything not on the allow list is refused rather than asked about. The only Bash commands allowed were the task's green command and its verbose form; file edits were limited to the working copy, and edits to tasks/, where the acceptance test lives, were denied. Each attempt's start-up record in the transcripts shows the six tools, no MCP servers and the dontAsk mode. The bench hashed tasks/ before and after every attempt, and no attempt changed it.

• Caps and timing. 30 turns and $2.00 (Claude Code's own estimate) per attempt, and a 900-second wall-clock timeout that kills the agent's whole process tree. Time-to-green is as defined below: from launching the agent to the agent exiting, with the green command run by the bench straight after. Attempts ran one at a time, with tasks interleaved and the model that went first alternating, so that an early stop would still have left a balanced sample.
• Tool calls were counted from the stream-json transcript, the addition the runner section below says JSON mode lacks. The driver that ran the attempts is not published, because it holds paths from our machine; before the full run we checked on T04 that the shipped runner drives Claude Code the same way through --agent-cmd.

The data and the script

• time-to-green-results-2026-10-30.csv: 60 rows, one per attempt, in the task file's fourteen columns, with the run id, token counts and refusal count in notes.
• time-to-green-token-ledger-2026-10-30.csv: every token count and the cost estimate per attempt, appended as each attempt finished, with running totals.
• time-to-green-transcripts-2026-10-30.json, about 1.2 MB: the redacted transcripts of all 60 attempts, one per line. They keep what the agent wrote, every tool call with its input and result, every refusal, and the counts from the result object. We removed session and request ids, account rate-limit events and the list of what else is installed on the machine, and replaced local paths with <workdir>.
• time-to-green-results.py reads the two CSVs, checks that they agree attempt by attempt, and prints every number in this section. This is its output, unedited:

$ python time-to-green-results.py
attempts: 60 (10 tasks x 3 repeats x 2 models); Claude Code 2.1.314
window: 2026-10-30T18:42:57+00:00 to 2026-10-30T19:12:46+00:00; agent time 1,763 s
outcomes: {'green': 60}; tasks/ changed in 0 attempts; cost basis ['list']

model | attempts | green | time-to-green median (IQR) s | fastest-slowest s | median turns | median tool calls | median tokens | median est. $ | est. $ per green
claude-opus-5-5 | 30 | 30 | 32.4 (26.4-40.7) | 18.9-56.4 | 9 | 8 | 70,876 | 0.145 | 0.154
claude-sonnet-5-5 | 30 | 30 | 25.6 (19.6-29.8) | 10.7-40.2 | 8 | 7 | 50,810 | 0.051 | 0.053
all | 60 | 60 | 28.6 (23.3-35.2) | 10.7-56.4 | 9 | 8 | 63,319 | 0.093 | 0.104

tokens | uncached input | cache write | cache read | output | total | cache-read share | est. $ (list)
claude-opus-5-5 | 368 | 296,623 | 1,935,167 | 93,580 | 2,325,738 | 83% | 4.63
claude-sonnet-5-5 | 302 | 211,123 | 1,336,524 | 61,046 | 1,608,995 | 83% | 1.59
all | 670 | 507,746 | 3,271,691 | 154,626 | 3,934,733 | 83% | 6.22

permission refusals | refusals | attempts with one or more
claude-opus-5-5 | 41 | 30 of 30
claude-sonnet-5-5 | 25 | 25 of 30
all | 66 | 55 of 60

caps (30 turns, $2.00, 900 s) | most turns 14 | highest est. $0.216 | slowest 56.4 s

task | claude-opus-5-5: three attempts s (median) | claude-sonnet-5-5: three attempts s (median)
T01 | 56, 40, 43 (42.9) | 33, 21, 30 (29.7)
T02 | 36, 29, 28 (29.1) | 15, 30, 19 (18.6)
T03 | 23, 49, 26 (25.5) | 29, 34, 16 (28.6)
T04 | 41, 26, 32 (31.5) | 26, 16, 33 (25.5)
T05 | 21, 22, 19 (20.9) | 25, 25, 11 (25.2)
T06 | 42, 44, 33 (42.0) | 40, 40, 30 (39.6)
T07 | 29, 35, 26 (28.6) | 27, 20, 26 (25.7)
T08 | 43, 29, 41 (40.6) | 30, 36, 20 (29.9)
T09 | 23, 27, 33 (27.4) | 20, 15, 16 (16.4)
T10 | 43, 31, 38 (38.3) | 25, 27, 26 (25.9)
faster median, tasks won: claude-opus-5-5 2, claude-sonnet-5-5 8

Limits

• One machine (Windows, Python 3.13.12), half an hour on one afternoon, one Claude Code version, two models and three attempts per task. Medians over 30 attempts per model are steady enough to compare; per-task medians of three are not.
• The tasks are small and fully specified, too small to separate the models on pass rate. A 100% pass rate here says nothing about larger or vaguer work.
• The acceptance tests could be read, though not edited, and in some attempts the agent opened them. That is what this page's method specifies, and it makes these tasks easier than a benchmark with hidden tests.
• The tests are ours. Another author could resolve the open choices differently.
• There was no operating-system network block. Containment was Claude Code's permission rules plus the hash check of tasks/ afterwards, and python -m unittest runs whatever test code is in the repo.
• The runs used a subscription login. The dollar figures are list-price estimates, and the seconds are that afternoon's response times on that login; we did not repeat the run with an API key.
• The permission rules shaped what the agents did. A looser rule set would probably mean more turns and more time.

Why time-to-green and not pass rate

A pass rate tells you whether an agent got there. It does not tell you how long you waited, how much it cost, or whether the passing run was one you would have merged. The AgentLens paper (arXiv:2605.12925) studied 2,614 OpenHands trajectories on 60 SWE-bench Verified tasks and reports that among passing trajectories "10.7% exhibit behavior we call a Lucky Pass", with lucky rates ranging "from 0.5% to 23.2%" across models. A pass, on its own, is a coarse signal.

Time carries information a pass does not. The Fail-Fast, Restart-Smart paper (arXiv:2608.03222) opens from the observation that "failed runs tend to be longer and exhibit redundant exploration or looping", and builds an early-termination monitor on it. If failures are long, then the distribution of time-to-green across repeated attempts on the same task is a shape worth recording, not just its mean.

Time-to-green is the interval from the moment a prompt is submitted to the first moment a named check exits 0. It is cheap to measure, it is what a developer feels while waiting, and it is honest in the way a benchmark score is not: a task either went green under a fixed budget or it did not.

The definition, precisely

Each task in the dataset has three parts: a repository state, a one-line goal, and a green condition. The green condition is always a command, and always a test module that must exit 0. Three rules make the number comparable across attempts:

1. Start is when the prompt is submitted. Time spent writing the prompt is not counted; the prompt text is fixed per task and lives in the task column.
2. Green is the first time the green command exits 0 with no edits to the acceptance tests. The acceptance test file is written before the run and is read-only for the agent, enforced with a deny rule on its path, not with an instruction.
3. Budget caps every attempt the same way. For Claude Code the CLI reference documents --max-turns ("Limit the number of agentic turns (print mode only). Exits with an error when the limit is reached") and --max-budget-usd ("Maximum dollar amount to spend on API calls before stopping (print mode only). Spend from subagents counts toward the cap"). The runner's default is 30 turns and 2.00 USD.

An attempt ends in one of four outcomes: green, red (the agent exited on its own and the check still fails), timeout (the runner's wall clock expired), or agent_error (non-zero exit from the agent process, budget exhaustion included). A red attempt is data, not a failure of the method.

The ten tasks

Five are on the tally CSV tool from A one-paragraph spec for a coding agent, five on the logreport script from the legacy-code tutorial. They were chosen to span the kinds of small work an agent gets in practice: a new flag, a new subcommand, a parsing rule, a bug fix under a pinned test, an output format.

T06 is the interesting one. The legacy script ships with a characterization test that pins the bug (see the first tests in the refactoring tutorial). Before the T06 run, that one test is replaced by the corrected expectation, so the suite starts red for exactly one reason. That is the cleanest possible task: one failing test, one named cause.

The acceptance test modules named in the green column are the pre-run step the method requires. The first version of this page did not ship them; the ten we wrote for the 30 October run are published now (see Method). For T01 the core is a call to main(["stats", path, "--col", "price", "--json"]) followed by json.loads on the captured stdout and a check on the seven keys, with the shipped suite run alongside as regression.

The runner

The shipped script does four things: look up the task row, run the agent command with the task text substituted for {prompt}, run the green command, and append a row. The agent command is a template so any harness can be timed; the default is Claude Code in print mode with JSON output.

DEFAULT_AGENT = 'claude -p "{prompt}" --output-format json --max-turns 30 --max-budget-usd 2.00'

JSON output is what makes the cost column possible. The non-interactive page says that with --output-format json "the response payload includes total_cost_usd and a per-model cost breakdown", and the same page calls both "client-side estimates" that "can differ from your actual bill". The cost-tracking page adds a caveat the runner respects by reading total_cost_usd rather than usage: the usage field "counts only the top-level agent loop, so tokens consumed inside subagents are not added", while total_cost_usd includes subagent requests. The runner also copies a num_turns field into the turns column when the payload carries one; the docs page we read names the cost fields, not that one, but the result object carried num_turns in all 60 attempts of the 30 October run. Tool calls are left blank in JSON mode; they need stream-json and a count over tool_use blocks, which is how the 30 October run counted them.

The core of the timing:

def run_agent(cmd, timeout):
    t0 = time.monotonic()
    try:
        proc = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
    except subprocess.TimeoutExpired:
        return {}, "timeout", time.monotonic() - t0
    elapsed = time.monotonic() - t0
    ...

Note what is being timed: the agent process, end to end. The green check runs after and is not included, because a slow test suite is a property of the repository, not the agent.

What ran when the protocol was written

Two stand-in runs, with a Python one-liner in place of an agent so that no API was called and nothing was edited. This proves the plumbing and nothing else.

The first uses the shipped tally test suite as the green command, so the check passes:

$ python time-to-green-ten-small-tasks-method.py --task T01 --harness stand-in \
    --agent-cmd "python -c \"import json;print(json.dumps({'total_cost_usd':0.0,'num_turns':0}))\"" \
    --green-cmd "python -m unittest build-a-cli-tool-with-an-agent-from-a-spec_test.py"
{"task_id": "T01", "outcome": "green", "seconds_to_green": "0.1", "turns": 0,
 "cost_usd": 0.0, "notes": "agent_elapsed_s=0.1; green_exit=0; api_ms="}
exit=0

The second uses a command that exits 1 as the green check, and a stand-in that prints non-JSON, to exercise the red path and the JSON fallback:

$ python time-to-green-ten-small-tasks-method.py --task T06 --harness stand-in \
    --agent-cmd "python -c \"print('not json')\"" --green-cmd "python -c \"import sys;sys.exit(1)\""
{"task_id": "T06", "outcome": "red", "seconds_to_green": "", "turns": "",
 "cost_usd": "", "notes": "agent_elapsed_s=0.1; green_exit=1; api_ms="}
exit=1

Both rows landed in the results CSV with the same fourteen columns as the dataset. The 0.1 seconds is the cost of starting a Python interpreter, and it is labelled stand-in in the harness column so it can never be mistaken for a measurement.

The task file

The shipped CSV is a task file, not a dataset of results: it holds the ten task definitions, their green commands and the column layout a run fills in.

_README  time-to-green-ten-small-tasks-method.csv
task_id          T01..T10
task             the exact prompt text submitted, fixed per task
green_command    the check that defines green; must exit 0
harness          agent runner name, e.g. Claude Code, Codex CLI, mini-swe-agent
harness_version  the version string reported by the tool at run time
model            model id as configured for the run
started_at       ISO 8601 UTC, prompt submission
green_at         ISO 8601 UTC, first green; empty if never green
seconds_to_green agent wall time in seconds; empty if never green
turns            num_turns from the JSON result, when the harness reports it
tool_calls       count of tool_use blocks (stream-json only); empty otherwise
cost_usd         total_cost_usd from the JSON result; a client-side estimate
outcome          green | red | timeout | agent_error (+green if the check passed anyway)
notes            repo state and any deviation from the method

The task file still ships as ten rows with the task, green command and notes columns filled and every result column (harness through outcome) empty, because it is the runner's input. The 30 October results are a separate file in the same fourteen columns, time-to-green-results-2026-10-30.csv, one row per attempt. A row with a number in it always carries a harness version and a model, and the runner writes those from its flags rather than inferring them.

What would make the numbers mean something

Three attempts per task per harness, not one, because the FailFast paper's point about long failures implies a heavy tail. The same machine, the same repository state (a fresh worktree per attempt, so nothing leaks between runs), and the same budget flags. The acceptance tests hidden from edits. And a red outcome recorded with the same care as a green, because the ratio between them at a fixed budget is the number people actually want when they ask whether an agent is worth it for small tasks. The runner does not enforce any of that; it records what you tell it. The discipline is yours. The 30 October run kept to it: three attempts per task and model, one machine, a fresh copy of the repo for every attempt, the same caps, and edits to the acceptance tests denied.

Correction, 29 October 2026: this page's title asked how to measure the ten tasks, its description called the shipped CSV "a first dataset", and its opening line promised "a measurement you can run yourself", which read as a promise of results. The CSV holds the ten task definitions, their green commands and the column layout for results; no agent run has been recorded, so every result column is empty, and the acceptance test modules the green commands name are not shipped yet. The title, the description, the opening and the section on the CSV now say so. The rest of the article is unchanged.

Update, 2 November 2026: the first results are in. A new section gives 60 attempts of Claude Code from 30 October: time to green, turns, tool calls, tokens, estimated cost and permission refusals, with the method and its limits. The ten acceptance tests and the T06 and T07 test swaps are published, with a script that rebuilds the starting repos, and so are the per-attempt results, the token ledger, redacted transcripts and the script that recomputes every number. The title, description and opening now say so, and the sections on the acceptance tests, the runner and the task file are updated to match.