Measurements · 2026-09-05 · 1,366 words · 6 min read

Coding agent time-to-green: how do you measure ten small tasks fairly?

A method and a first dataset for timing a coding agent from prompt to first passing check: ten defined tasks, a runner script, and only honestly filled columns.

measurement time-to-green dataset claude code benchmark method

You get a measurement you can run yourself: ten small, fully specified coding tasks, a definition of "green" that a script can check, a runner that times an agent from prompt to first green and writes one CSV row per attempt, and the empty dataset those rows go into. What you do not get is results. No agent ran in this session, and the measurement columns in the shipped CSV are blank on purpose. What ran here was Python 3.13.12 on Windows: the runner, twice, with a stand-in command in place of an agent, to prove the timing and CSV path work.

Why time-to-green and not pass rate

A pass rate tells you whether an agent got there. It does not tell you how long you waited, how much it cost, or whether the passing run was one you would have merged. The AgentLens paper (arXiv:2605.12925) studied 2,614 OpenHands trajectories on 60 SWE-bench Verified tasks and reports that among passing trajectories "10.7% exhibit behavior we call a Lucky Pass", with lucky rates ranging "from 0.5% to 23.2%" across models. A pass, on its own, is a coarse signal.

Time carries information a pass does not. The Fail-Fast, Restart-Smart paper (arXiv:2608.03222) opens from the observation that "failed runs tend to be longer and exhibit redundant exploration or looping", and builds an early-termination monitor on it. If failures are long, then the distribution of time-to-green across repeated attempts on the same task is a shape worth recording, not just its mean.

Time-to-green is the interval from the moment a prompt is submitted to the first moment a named check exits 0. It is cheap to measure, it is what a developer feels while waiting, and it is honest in the way a benchmark score is not: a task either went green under a fixed budget or it did not.

The definition, precisely

Each task in the dataset has three parts: a repository state, a one-line goal, and a green condition. The green condition is always a command, and always a test module that must exit 0. Three rules make the number comparable across attempts:

1. Start is when the prompt is submitted. Time spent writing the prompt is not counted; the prompt text is fixed per task and lives in the task column.
2. Green is the first time the green command exits 0 with no edits to the acceptance tests. The acceptance test file is written before the run and is read-only for the agent, enforced with a deny rule on its path, not with an instruction.
3. Budget caps every attempt the same way. For Claude Code the CLI reference documents --max-turns ("Limit the number of agentic turns (print mode only). Exits with an error when the limit is reached") and --max-budget-usd ("Maximum dollar amount to spend on API calls before stopping (print mode only). Spend from subagents counts toward the cap"). The runner's default is 30 turns and 2.00 USD.

An attempt ends in one of four outcomes: green, red (the agent exited on its own and the check still fails), timeout (the runner's wall clock expired), or agent_error (non-zero exit from the agent process, budget exhaustion included). A red attempt is data, not a failure of the method.

The ten tasks

Five are on the tally CSV tool from Build a CLI tool with an agent from a one-paragraph spec, five on the logreport script from the two legacy-code tutorials. They were chosen to span the kinds of small work an agent gets in practice: a new flag, a new subcommand, a parsing rule, a bug fix under a pinned test, an output format.

T06 is the interesting one. The legacy script ships with a characterization test that pins the bug (see Add a test suite to a legacy script). Before the T06 run, that one test is replaced by the corrected expectation, so the suite starts red for exactly one reason. That is the cleanest possible task: one failing test, one named cause.

The acceptance test modules named in the green column are the pre-run step the method requires and are not shipped. Writing them is the next piece of work, and each is a few assertions in the style of the shipped suites. For T01, the whole acceptance test is a call to main(["stats", path, "--col", "price", "--json"]) followed by json.loads on the captured stdout and an equality check on seven keys.

The runner

The shipped script does four things: look up the task row, run the agent command with the task text substituted for {prompt}, run the green command, and append a row. The agent command is a template so any harness can be timed; the default is Claude Code in print mode with JSON output.

DEFAULT_AGENT = 'claude -p "{prompt}" --output-format json --max-turns 30 --max-budget-usd 2.00'

JSON output is what makes the cost column possible. The non-interactive page says that with --output-format json "the response payload includes total_cost_usd and a per-model cost breakdown", and the same page calls both "client-side estimates" that "can differ from your actual bill". The cost-tracking page adds a caveat the runner respects by reading total_cost_usd rather than usage: the usage field "counts only the top-level agent loop, so tokens consumed inside subagents are not added", while total_cost_usd includes subagent requests. The runner also copies a num_turns field into the turns column when the payload carries one; the docs page I read names the cost fields, not that one, so treat it as best effort until a real run confirms it. Tool calls are left blank in JSON mode; they need stream-json and a count over tool_use blocks, which is a later addition.

The core of the timing:

def run_agent(cmd, timeout):
    t0 = time.monotonic()
    try:
        proc = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
    except subprocess.TimeoutExpired:
        return {}, "timeout", time.monotonic() - t0
    elapsed = time.monotonic() - t0
    ...

Note what is being timed: the agent process, end to end. The green check runs after and is not included, because a slow test suite is a property of the repository, not the agent.

What ran here

Two stand-in runs, with a Python one-liner in place of an agent so that no API was called and nothing was edited. This proves the plumbing and nothing else.

The first uses the shipped tally test suite as the green command, so the check passes:

$ python time-to-green-ten-small-tasks-method.py --task T01 --harness stand-in \
    --agent-cmd "python -c \"import json;print(json.dumps({'total_cost_usd':0.0,'num_turns':0}))\"" \
    --green-cmd "python -m unittest build-a-cli-tool-with-an-agent-from-a-spec_test.py"
{"task_id": "T01", "outcome": "green", "seconds_to_green": "0.1", "turns": 0,
 "cost_usd": 0.0, "notes": "agent_elapsed_s=0.1; green_exit=0; api_ms="}
exit=0

The second uses a command that exits 1 as the green check, and a stand-in that prints non-JSON, to exercise the red path and the JSON fallback:

$ python time-to-green-ten-small-tasks-method.py --task T06 --harness stand-in \
    --agent-cmd "python -c \"print('not json')\"" --green-cmd "python -c \"import sys;sys.exit(1)\""
{"task_id": "T06", "outcome": "red", "seconds_to_green": "", "turns": "",
 "cost_usd": "", "notes": "agent_elapsed_s=0.1; green_exit=1; api_ms="}
exit=1

Both rows landed in the results CSV with the same fourteen columns as the dataset. The 0.1 seconds is the cost of starting a Python interpreter, and it is labelled stand-in in the harness column so it can never be mistaken for a measurement.

The dataset

_README  time-to-green-ten-small-tasks-method.csv
task_id          T01..T10
task             the exact prompt text submitted, fixed per task
green_command    the check that defines green; must exit 0
harness          agent runner name, e.g. Claude Code, Codex CLI, mini-swe-agent
harness_version  the version string reported by the tool at run time
model            model id as configured for the run
started_at       ISO 8601 UTC, prompt submission
green_at         ISO 8601 UTC, first green; empty if never green
seconds_to_green agent wall time in seconds; empty if never green
turns            num_turns from the JSON result, when the harness reports it
tool_calls       count of tool_use blocks (stream-json only); empty otherwise
cost_usd         total_cost_usd from the JSON result; a client-side estimate
outcome          green | red | timeout | agent_error (+green if the check passed anyway)
notes            repo state and any deviation from the method

Ten rows are shipped. The task, green command, and notes columns are filled; every measurement column is empty because no measurement was taken. A row with a number in it will always carry a harness version and a model, and the runner writes those from its flags rather than inferring them.

What would make the numbers mean something

Three attempts per task per harness, not one, because the FailFast paper's point about long failures implies a heavy tail. The same machine, the same repository state (a fresh worktree per attempt, so nothing leaks between runs), and the same budget flags. The acceptance tests hidden from edits. And a red outcome recorded with the same care as a green, because the ratio between them at a fixed budget is the number people actually want when they ask whether an agent is worth it for small tasks. The runner does not enforce any of that; it records what you tell it. The discipline is yours.