I’ve written before that you shouldn’t trust what a model says — only what a mechanical check can show. A test that genuinely fails before a fix and passes after is a fact the model can’t fabricate by being confident. That still holds. This essay is about the failure mode hiding one level underneath it, the one I was slower to learn to distrust because it’s better disguised: the mechanical check itself can be wrong, and when it is, it is more convincing than a confident model — because it wears the costume of rigour.
Working out where mechanical verification ends and judgment has to take over — in production, not in theory.