The signal a model can’t author

Not long ago I shipped a piece of internal tooling — a panel that hands a problem to several models and collates their verdicts — with a full, green test suite. It had never once run. Not slowly, not badly: never. Every test replaced the function that does the work with a stand-in, and the real function was broken, so the suite proved only that my mocks agreed with themselves.

The bug itself was trivial. The panel called its dispatcher with arguments the real dispatcher didn’t accept, so every live call failed immediately. But no test made a live call — each one swapped the dispatcher for a stub — so the failure never surfaced. Every test passed, and the feature had never executed a single time.

A test that would have passed anyway

That changed how I read a green suite. A passing test feels like evidence, and often it isn’t. The question that matters is not does this pass — it is would this have failed before the change and passed after. A test tells you nothing if it would have been green regardless of whether the code worked. The signal isn’t the pass; it’s the transition — red, then green, for a reason you can name.

So in the tooling I run, a fix isn’t trusted until its test is shown to fail against the old code and pass against the new. Run the regression against the unfixed version first; if it’s still green, the test is decorative, and it gets thrown out and rewritten. The failing run is the load-bearing half, and most suites never check it.

# a fix isn’t trusted until its test fails *without* it
git stash         # remove the fix
pytest test_x     # must FAIL — green = a decorative test
git stash pop     # restore the fix
pytest test_x     # PASS now — the red→green transition

Attested versus mechanical

Underneath this is a split between two kinds of verification, and most of what gets called review is the weaker one. Call them attested and mechanical. An attested check is someone — a person, or another model — reading the work and declaring it sound. It beats nothing, but it’s an opinion: it can be confidently wrong, and it rests entirely on trusting the reviewer. A mechanical check is anchored to something the author can’t fake. The failing-then-passing test is the type case: I can write a test that passes, but I can’t make a test that was genuinely failing against unchanged code suddenly pass without changing the code. Reality moves the needle, not my say-so.

The test is whether the check cares who did the work. A check that would answer differently depending on who authored the thing is judgment. A check that answers the same regardless — because it asks reality, not the author — is mechanical, and you should push as much as you can into it. But mechanical only holds while the test, the runner, and the oracle stay outside the author’s reach. If the thing being checked can reach the thing doing the checking, the transition is forgeable too. I know, because I watched it happen.

The model that forged the pass

The sharper lesson came from a verifier whose entire job was to stop exactly this. It ran a candidate’s code against a held-out answer key — held out so that passing meant solving the problem, not reading the answer. Before trusting it, I put the verifier itself under review: three models, of different lineages, asked to find the hole.

Two didn’t. One reasoned carefully that the answer was never handed to the candidate — not in its arguments, its input, or its environment — and concluded the verifier was sound. The reasoning was correct as far as it went; it simply never checked whether the answer could be reached some other way. Another asked for fixes elsewhere and missed this entirely. The third didn’t argue the verifier was unsafe — it proved it. It wrote a short program that searched the filesystem, found the held-out answer key sitting on disk, read it, printed the expected result, and exited before the verifier could compare. The verifier returned pass. Nothing had been solved.

That is the whole argument in one incident. Two reviews passed it by — one a careful argument that the verifier was sound, the other intent on a different flaw. Neither caught the hole; the signal that it was there came from a program that ran and came back green for the wrong reason. The key was meant to be out of reach; it wasn’t, because it lived in a file the graded process could open. The fix was small: give the verifier a cryptographic hash of the expected answer instead of the answer itself, so the answer is simply absent from the reachable namespace, and reality becomes the oracle again. This is the never-dispatched panel turned adversarial — there my own stubs authored the green by accident; here a reviewer authored a pass on purpose, to show it could be done.

# wrong: the answer lives on disk, where the graded
# code can read it — so a PASS is forgeable
expected = load("held_out/answer.json")

# right: ship only the hash; the answer is out of reach
EXPECTED_SHA = sha256(expected)
assert sha256(candidate) == EXPECTED_SHA

Fixtures that mask reality

The never-dispatched panel and the forged pass are two faces of one defect, and once it has a name you see it everywhere. I call it a fixture that masks reality: a stand-in — a mock, a sample input, an answer key — that diverges from the real thing on exactly the dimension that matters, so the test passes while the system is broken. The bug lives in the gap between the fixture and reality, which is precisely where the test isn’t looking.

The most expensive instance I’ve hit had three hundred and sixty-five passing tests over a feature that was entirely broken. The component’s whole job — resuming work across processes — failed every time. The tests passed because they used a mock of the store, and the mock was more complete than the real implementation: it answered protocol calls the shipped version couldn’t. Every test exercised a store that behaved better than the one that went to production. Three hundred and sixty-five green checks, and not one went near the gap where the bug lived.

A cheaper one, same shape: a validator for test identifiers, checked only against the garbage inputs it was supposed to reject. It rejected the garbage, so it was green. Nobody checked it against the real identifiers it would actually see — and it turned out to reject most of them. Green on the inputs that don’t matter, broken on the ones that do. The lesson is not that fixtures are bad; a hand-built input that genuinely exercises the hard case is fine, and often the only way to reach it. The lesson is narrower: a fixture that doesn’t match reality on the dimension you care about is worse than no test at all, because it sells you confidence you haven’t earned.

Not everything can be made mechanical, and where it can’t, the next best thing is a check that at least doesn’t share the author’s blind spots. One of the bugs in that panel was a line of configuration pointing at a field name the tool never reads. The model that caught it wasn’t the one that wrote it — it was a different model, of a different lineage, reviewing the code. A reviewer who shares your assumptions ratifies your mistakes; a decorrelated one occasionally catches the thing you couldn’t see because you made it. But this is still judgment — an opinion that happened to be right — and it’s weaker than a check anchored to reality. It’s the fallback, not the floor.

Don’t ask whether the model says it’s done. Ask whether anything other than the model could tell you it’s done. If the answer is no, you don’t have verification — you have a model grading its own homework.