Naming is the easy part

I wrote a while back about a kind of bug I’d learned to recognise: a fixture that masks reality — a stand-in, a mock or a sample input, that diverges from the real thing on exactly the dimension you’re testing, so the suite stays green while the system is broken. Naming it felt like the win. An un-named bug you rediscover every time; a named one, I assumed, you could hunt. So I tried to make the name do some work — to turn it into a standing check my reviewers would apply. That is where I learned the naming was the easy part.

The trouble showed up the moment I tried to write the check down. “Did you check for a fixture that masks reality?” is not a question anyone can act on. The name covered a validator that was only ever tested on the junk it was built to reject, a control that only exercised the safe path, and a verifier that recomputed its proof from a file the caller could quietly edit. Those are not one bug. They share a shape — the test agreed with a stand-in instead of reality — but a shape is not a test. A name you cannot turn into something a reviewer runs is not a tool. It is a feeling.

And the single name is seductive precisely because it feels like understanding. Several messy incidents collapse into one category, and the collapse reads as insight — you’ve found the deep thing they have in common. But the thing they have in common is a description, not a remedy. The validator and the fake engine and the editable file all let a test agree with something that wasn’t real; knowing that is satisfying and buys you nothing at three in the afternoon when a reviewer has to decide whether this particular diff is safe. The comfort of the category is exactly what makes it useless as a check.

The costumes don’t share a fix

Put the cases side by side and the problem is plain: each one is caught by a different check. The validator that only ever saw the junk it should reject is caught by testing it on the real inputs it will actually meet. The control that proved the safe path is caught by forcing it down the path that was actually vulnerable. The verifier reading a caller-supplied tree is caught by recomputing from the git blob at the recorded commit, not a caller-supplied file on disk. One name, and already three different fixes.

The case that made it undeniable was a fake engine. We had tests for running several agents at once, and they passed. They passed because the stand-in engine started instantly — and the real engine took thirty to ninety seconds to start, blocking everything queued behind it the whole time. The parallelism the suite certified was a property the real system did not have. Nothing in the test was wrong, exactly; the fixture was simply cheaper than reality on the one axis — time — that the test existed to check. Another costume, and it needs a check of its own: test against the real cost, not a free stand-in.

# the stub the parallelism test trusted
class FakeEngine:
    def start(self): ...        # instant — free by construction

# the real one it stood in for
class Engine:
    def start(self):
        boot_runtime()         # 30–90s; blocks the slot behind it

Split until you can check

The fix for the name was not a better definition. It was to stop using the one name. The cases split cleanly into three, and each got its own bar — the specific thing a reviewer checks for. Verification anchored to state the caller can edit gets traced back to immutable ground truth. An adversarial control has to be shown to move the verdict on the channel that was actually vulnerable, not a neighbouring one. A test has to exercise the load-bearing case on real production input, and the fix has to be shown to fail against the old code before it passes against the new. Three classes, three checks, no shared instance between them. The value of a class name turned out to be exactly its checkability — and the broad, comforting name had none.

The rule that mattered was the one I’d skipped: no instance could count toward two classes. It sounds like bookkeeping, and it is, but it’s load-bearing bookkeeping — the moment a single failure can be filed under two headings, the headings have stopped carving the problem at its joints and started flattering the taxonomy. Disjoint, or it doesn’t count.

The over-fit caught me

I would like to say I split the name on purpose. I didn’t. I had promoted it — written it up as a single standing check and, in the commit, called it the strongest of the lot. What I had actually done was lump three different mechanisms under one tidy heading and, to make the taxonomy look clean, quietly let some of the same evidence count twice. The review panel caught it. It made me re-ground the thing into disjoint classes — no instance allowed to count toward two — and the umbrella name, the strong one I’d been pleased with, did not survive the fold. An essay about over-lumping, and its author got caught over-lumping — by the very review he’d set up to promote it. That is the most honest version of how this works: the name I was proudest of was the one doing the least, because it was carrying too much.

So the test I’d offer for any defect name you’re proud of is blunt: say the check. Not the description — the check. What does a reviewer actually run, or read, or compare, when they apply it? If you can answer that crisply, the name is real, and it will save you the rediscovery tax every time the bug comes back in a new costume. If you can’t — if the honest answer is that you sort of look for the vibe — then you haven’t named a class, you’ve named a mood, and it will hand you exactly the false confidence the bug it describes hands you.

A name you can’t check is just a feeling. Split it until each piece has a test a reviewer can run — or admit you’ve named nothing.