Workflows · 2026-09-05 · 1,295 words · 6 min read

Tests first with an agent: why must the test fail before it passes?

The red-green loop for agents: why a test that never failed proves nothing, hook-based gates, the lucky-pass numbers, and a script that enforces the order.

testing red-green hooks claude code unittest

What you get here: the argument for making an agent show you a failing test before it writes the code, the published rate at which passing runs turn out to be luck, two ways to enforce the order with the harness rather than with willpower, and a small gate script that ran in this session on a real module. The output is real and unedited.

The pass signal is the only honest one

An agent stops when the work looks done. The Claude Code best-practices page says it plainly: "Claude stops when the work looks done. Without a check it can run, 'looks done' is the only signal available, and you become the verification loop." A test suite is the cheapest check that closes that loop, because the agent can run it, read the exit code, and iterate without you.

But a green suite only means something if it could have been red. A test that the agent writes after the implementation, tuned until it passes, has never demonstrated that it detects anything. Worse, an agent that is told "add tests" will often write tests that call the code and assert on whatever the code returned. Those tests pass on the first run, which is exactly the case where you learned nothing. The red-green loop is older than agents, but agents make the red half load-bearing in a way it was not before: the failing run is the only moment when you know the test is connected to the behaviour you care about.

The AgentLens paper (arXiv:2605.12925) measured what happens when the pass signal is trusted on its own. The authors evaluated 2,614 OpenHands trajectories across eight model backends on 60 SWE-bench Verified tasks and found that "among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass", which they characterise as regression cycles, blind retries, missing verification, or temporally disordered work. Across models the lucky rate ranged from 0.5% to 23.2%, and "some models move by as many as five rank positions when ranked by quality score instead of pass rate." A patch that passes the hidden tests after four blind retries is still a pass on the leaderboard. In your repository it is a patch you cannot trust.

The loop, stated for an agent

The instruction I give is three sentences, and the order is the whole point:

1. Write one test for the behaviour in the spec. Run it. Show me the failure.
2. Make it pass with the smallest change that does so. Run it again. Show me the pass.
3. Run the full suite. Show me that nothing else changed.

Each "show me" is a tool result that lands in the transcript, so the evidence is there without me re-running anything. The best-practices page recommends exactly this: "Have Claude show evidence rather than asserting success: the test output, the command it ran and what it returned."

Python's own runner is enough for this. The unittest documentation describes a framework that "supports test automation, sharing of setup and shutdown code for tests, aggregation of tests into collections", and a single test can be addressed by id: python -m unittest test_module.TestClass.test_method. That addressability is what makes the red run cheap. You do not run the suite to see one failure; you run one test.

Enforcing the order with the harness

Willpower does not scale to a forty-turn session. The harness can hold the line instead.

The first option is a Stop hook. The best-practices page describes it as a deterministic gate: "a Stop hook runs your check as a script and blocks the turn from ending until it passes. Claude Code overrides the hook and ends the turn after 8 consecutive blocks." The eight-block ceiling is the harness protecting you from a hook that can never pass; it is also the number you should remember when the agent reports done but the suite is still red.

The second option is a PreToolUse hook that watches for the moment an implementation file is edited before any red run has been recorded. The hooks reference shows the JSON a PreToolUse hook receives on stdin, including tool_name, tool_input, and cwd, and states that "Exit 2 means a blocking error. On events that can block, exit 2 blocks whether or not you print JSON." A hook can also return a structured decision, permissionDecision of allow or deny, with a reason the model sees. The shipped script keeps a ledger file precisely so that a hook like this has something to read.

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          {
            "type": "command",
            "command": "python .claude/hooks/require-red.py"
          }
        ]
      }
    ]
  }
}

The hook script itself is a dozen lines: read the JSON from stdin, and if tool_input.file_path is not a test file and .redgreen.json has no entry with a red timestamp and no green, exit 2 with a message saying which test to write first. I have not shipped that hook here because its file-classification rule is project-specific; the gate script below is the reusable part.

The third option costs nothing to set up: the /goal command. The same best-practices page says a goal condition is re-checked "after every turn and Claude keeps working until the goal resolves." Setting the goal to "the full suite passes and the new test in the ledger has both a red and a green timestamp" makes the loop the session's exit condition.

The gate script

The shipped script has two commands. red <test id> runs one unittest id and refuses to record it unless the run fails or errors. green <test id> refuses to run at all unless a red was recorded, then requires that test to pass, then runs python -m unittest discover over the folder and requires that to pass as well. Both timestamps go into .redgreen.json.

I exercised it in this session on a two-function module in a temporary folder: a slugify function and three tests, one of which asked for a max_len parameter that did not exist yet. This is the real sequence and output, Python 3.13 on Windows, trimmed only for the long separator lines.

First, trying to skip ahead:

$ python tests-first-with-an-agent-red-green.py green test_slugify.SlugifyTests.test_truncates_to_max_length
GREEN GATE FAILED: no red run recorded for test_slugify.SlugifyTests.test_truncates_to_max_length. Run `red` first.
exit=1

Then the red run on the new test:

$ python tests-first-with-an-agent-red-green.py red test_slugify.SlugifyTests.test_truncates_to_max_length
RED OK: test_slugify.SlugifyTests.test_truncates_to_max_length fails as required (exit 1). Recorded in .redgreen.json.
        self.assertEqual(slugify("one two three four", max_len=7), "one-two")
    TypeError: slugify() got an unexpected keyword argument 'max_len'
    Ran 1 test in 0.000s
    FAILED (errors=1)
exit=0

A red run on a test that already passes is rejected, because it would prove nothing:

$ python tests-first-with-an-agent-red-green.py red test_slugify.SlugifyTests.test_basic
RED GATE FAILED: test_slugify.SlugifyTests.test_basic already passes, so it proves nothing yet.
    Ran 1 test in 0.000s
    OK
exit=1

After adding the max_len parameter, the green run passes the single test and then the whole suite of three:

$ python tests-first-with-an-agent-red-green.py green test_slugify.SlugifyTests.test_truncates_to_max_length
GREEN OK: test_slugify.SlugifyTests.test_truncates_to_max_length passes and the full suite passes. Ledger: red 2026-09-05T23:35:07+00:00 -> green 2026-09-05T23:35:07+00:00.
    Ran 3 tests in 0.000s
    OK
exit=0

The ledger after that run is one entry with both timestamps. The same second for red and green is honest here because I edited the file by hand between the two commands; in an agent session the gap is the agent's implementation time, which is a number worth keeping. The time-to-green method uses exactly this pair of timestamps.

Where this breaks

Red-green assumes the test can be written before the code, which is true for behaviour you can name and false for exploratory work. When the task is "find out why the export is slow", there is no test to write first; write the spec as a question and come back to the loop once you know what the fix is.

It also assumes the red run fails for the right reason. The error above is a TypeError on a missing parameter, which is fine, but a test that fails because of a typo in the test itself also satisfies the gate. The script records the tail of the output so you can read why it was red; it does not judge it. Reading three lines of failure output is the human's part of the loop, and it is cheaper than reading the diff that a lucky pass produces.

Finally, the full-suite step catches regressions only for behaviour that already has tests. If the legacy code has none, start with characterization tests so that the suite has something to protect before the agent starts changing it.