When rigour lies

I’ve written before that you shouldn’t trust what a model says — only what a mechanical check can show. A test that genuinely fails before a fix and passes after is a fact the model can’t fabricate by being confident. That still holds. This essay is about the failure mode hiding one level underneath it, the one I was slower to learn to distrust because it’s better disguised: the mechanical check itself can be wrong, and when it is, it is more convincing than a confident model — because it wears the costume of rigour.

A green test that never ran the feature is a familiar problem. You learn to assert that the real path executed. But there’s a worse version, and it cost my system seventeen minutes of insisting on the exact opposite of the truth — seventeen minutes in which a correct fix stood “empirically disproven,” declared so twice, in calm and well-reasoned summaries, while the thing doing the disproving was the broken part.

The shape of the failure

The setup was mundane. A background process in one of my systems was suspected of leaking rows into a shared store — writing records it shouldn’t, or writing them after it was told to stop. A reviewer model found the mechanism and proposed a fix. The fix was correct. It was correct on arrival, in the first minute.

Then the verification ran, and reported that the fix also leaked. So the system kept going. For the next seventeen minutes the conclusion on the table was “the proposed fix is empirically disproven — it still leaks,” stated confidently, supported by a query against the store that returned rows. A second pass confirmed it. The reasoning was sound, the tone was rigorous, the evidence was right there in the database.

The evidence was contamination. The check asserted cleanliness by counting rows whose names matched a pattern — a pattern that every run, ever, had written into the same namespace. So the question it actually answered was not “did this run leak?” but “are there any rows from any run, ever, still sitting in the store?” There were — left by a run fifty minutes earlier, before the fix existed. The clean run, the one with the correct fix, had left exactly zero. The instrument counted history and called it the present.

A run-tag existed that could have isolated the measurement to this run’s output. But it scoped the cleanup, not the verdict. The delete knew which run it was; the pass/fail didn’t. So the fix kept being measured against the residue of the bug it had already fixed, and kept “failing.”

Why this is worse than a green lie

A test that passes without running the feature is dangerous because it’s silent. This is dangerous because it’s loud. It doesn’t quietly let something through — it actively asserts a falsehood, and it asserts it in the register you’ve trained yourself to trust.

I have spent a lot of effort teaching myself and my system to distrust confidence. A model that says “done” gets no credit; I want the test. But this failure inverts the defence: it arrives as the test. It is empirical. It read the database. It came back with a number. Every instinct I’d built to privilege the mechanical signal over the confident assertion was now aimed squarely at the wrong answer, because the mechanical signal was the thing that was broken — and it was broken in a way that produced not noise but a clean, specific, repeatable false negative.

A confident wrong model you can catch by asking for the check. But what catches a confident wrong check? “It’s empirically disproven” is a sentence that ends most arguments. It nearly ended this one: the verdict stood for seventeen minutes, and it would have stood longer if I hadn’t kept insisting — against a confident, specific, repeatable number — that the fix looked right and the measurement looked suspect, which, in the moment, sounds exactly like someone refusing to accept evidence.

That’s the trap in one line: a broken instrument reporting “disproven” is more persuasive than a green test, because it feels like rigour. The feeling of rigour is not rigour. Reading a store and getting a number is not the same as measuring the thing you meant to measure, and the gap between them is invisible precisely when the number is confident and specific.

The same failure with a different face

It wasn’t only that once. The same week, a different check in a different system did the same thing in a different costume.

A content gate stripped three claims about electrical work — claims that overstated when a homeowner could self-certify versus when the work was legally notifiable. The gate was right to strip them; they collapsed a legal conditional into an absolute. The signal was correct. But the explanation attached to the signal — the stated reason the gate had fired — was that the relevant regulations “aren’t in the corpus.” That explanation was false. The regulations were in the corpus; the correctly-phrased, conditional version of the guidance verified at high confidence, with a citation, when anyone bothered to check.

One question from me — “are you sure we didn’t ingest those regulations?” — overturned a conclusion that was one decision away from being written into policy. And the wrong decision would have permanently mis-handled legitimate guidance, on the strength of a correct signal with a false story attached to it.

The structural lesson is the same as the contaminated count, moved up a level. There, the measurement was wrong while the path ran fine. Here, the path was fine and the narrative about why was wrong. In both cases the underlying system did its job; what failed was the layer that turns a signal into a conclusion — corrupted once by measuring the wrong thing, once by summarising the right signal into the wrong story. A green or red signal is only as good as the explanation attached to it — and the explanation is where the unverified assumption hides once the path itself is sound.

There was even a tell, sitting in the data before any argument started. The gate’s own verdict vocabulary distinguished “outside our corpus” from “relevant material present, couldn’t confirm as worded.” The verdicts were the second kind. The contract had already said “in scope, phrasing failed” while the summary on top of it said “out of scope.” The structured signal carried a distinction the paraphrase flattened away — and the flattening was where the error entered. When a signal draws a fine distinction, read the distinction it actually drew before you summarise it into a coarser one. The summary is where you lie to yourself.

What it costs you to fix

The repair isn’t subtle once you see it, which is the frustrating part. The contaminated count needed the verdict scoped to state this run created — a unique tag in the assertion, not just the cleanup — or a purge proven to zero before the run starts. The false explanation needed one question asked of the corpus instead of assumed about it. Neither is hard. Both were skipped because the result looked finished: a number came back, a reason was given, the shape of rigour was present.

So the rule I’m trying to hold myself to is narrow and specific, and I’d rather state it plainly than dress it up: a pass/fail verdict has to be scoped to state this run created. In my case the run-tag already existed — I’d used it for the cleanup but not for the verdict, which is exactly the mistake to look for. If the assertion reads a store, a queue, a namespace, a log — anything that persists across runs — then “is it clean?” is a question about accumulated history, not about the change under test. A confident empirical refutation that depends on reading shared mutable state deserves exactly as much suspicion as a green test that never asserted the path ran. Both feel like evidence. Neither is, until you’ve isolated what they actually measured.

And one clean reading is not reliability. The contaminated count started with a single observation against dirty state — one query, one number, one wrong conclusion. The honest claim isn’t “it passed”; it’s “it passed N times from a baseline I proved was zero.” A fix that passes once may be passing on residue, or timing, or luck, and I try not to let the relief of having finally cracked it collapse the N back into a 1.

So I don’t get to conclude “and now my instruments are trustworthy.” I get to conclude something smaller and less satisfying: that rigour-shaped reasoning is not self-validating, that an empirical refutation is still a measurement and measurements can be wrong, and that the place to point my suspicion is precisely where I feel most certain — because certainty backed by a number is the most expensive kind to be wrong about.

The reviewer was right in the first minute; it took seventeen more for the instrument to stop drowning that out — and what stopped it wasn’t a better check, it was a person refusing to accept the number. The lesson isn’t “trust the reviewer.” It’s that the thing wearing the lab coat is not automatically the scientist — and the day it tells you, with great confidence and a number to back it, that the correct answer is disproven, that confidence is the thing to check first.