Decorrelation isn’t free
Not long ago, on a routine three-model review, the seat that caught the real bug was the one I’d least have predicted. Not the deepest reasoner of the three — a different model, of a different lineage, that happened to look where the other two didn’t. The defect was a check that would have blocked a worker the moment it started; two seats read straight past it, and the third stopped on it. The win wasn’t a sharper mind. It was a different blind spot.
That is the whole case for using more than one model to review, and it is narrower than it sounds. The value of the panel wasn’t a second opinion — opinions are cheap, and mostly they agree. It was that the seats failed in different places. I’ve watched it cut the other way too: on another review a different-lineage seat found a hole a deeper model had reasoned its way past, writing the exploit the other had argued couldn’t exist. Different seats win on different days, unpredictably. But all of it rests on one assumption — that the seats actually do fail differently — and that assumption is where almost everyone, including me, gets sloppy.
Decorrelation isn’t a roster
The word for what you want is decorrelation: the reviewers have different blind spots, so the thing one misses, another catches. The seductive error is to treat it as a property of the roster — we use a GPT, a Claude, and a Gemini, so we’re covered — when it is really a property of the failure modes. Different vendors is a hopeful proxy for different blind spots, not the thing itself. Models trained on overlapping slices of the same internet share the gaps that matter; point three of them at one problem and they can agree, fluently and confidently, on the same wrong answer. A vendor-diverse panel that happens to be blind in the same place is just an expensive way to feel covered.
Here the critic has a real point, and I’ll concede it plainly: you do not get a new guarantee for free by adding a differently-branded model. If the models share the blind spot, the panel inherits it. Vendor is a label on the box, not a measurement of what’s inside — and in my own setup that’s exactly what it’s recorded as: documentary, a note on which family a seat came from, not evidence that the seat fails differently from its neighbours.
The instrument that couldn’t measure itself
So I tried to measure it. If decorrelation is a claim about whether my seats fail differently, I ought to be able to check it rather than assume it. I started a log of disagreements — every time the seats split on a real review and the split got resolved, who was right and who was wrong. It looked like a decorrelation measure. It wasn’t, and the thing that showed me wasn’t a cleverer analysis: it was one of the seats, on a real review, pointing out the flaw.
The argument is simple once you see it, and it is fatal to the log as a measure. Every entry is a disagreement — a defect at least one seat caught. But the failure that actually justifies keeping a deep, decorrelated seat is the one the whole panel misses. A defect everyone misses never produces a disagreement, so it never enters the log. The log can only ever hold defects that were legible to someone; the never-caught set — the exact thing I wanted to measure — is absent from it by construction. I wrote that down, in the file itself: this is calibration data on legible defects, not a decorrelation oracle. The measure I built cannot see the thing it was built to measure.
I like this as a story more than I should: decorrelation caught the flaw in my attempt to measure decorrelation — the deep seat, doing exactly the catch-what-others-miss job that justifies it, caught it on the very instrument meant to justify it. But this essay is about not claiming evidence you don’t cleanly have, so I have to be careful with my favourite beat. That catch isn’t clean. It was adjudicated in the same session that was arguing about that seat, and the log flags it, in its own words, as correlated evidence, not clean. So I keep it as the best illustration I have and not as proof of anything. The evidence that decorrelation pays off, such as it is, lives in the other splits — the ones resolved by running the code, where the adjudication didn’t depend on anyone’s opinion.
What I enforce, and what I infer
There is one kind of decorrelation I actually enforce, and it’s worth separating from the kind I only hope for. When a check certifies a piece of work, I make the certifier a different model from the one that authored the work — chosen so its identity can’t equal the author’s, and if no decorrelated certifier is available, the thing refuses to run rather than quietly let a model bless its own output. That’s real, and it’s in code. But I should be exact about how strong it is: it holds inside the one gate that needs it, it defends against the accident of a model reviewing itself, and it is not proof against someone deliberately trying to defeat it — it’s enforced, not bulletproof. Outside that gate it’s a convention I keep, not a guarantee the system imposes.
That’s relational decorrelation — the certifier isn’t the author — and it’s a smaller, lesser claim than the one the panel actually rests on, which is that the seats have different blind spots. That second claim I neither enforce nor measure. I infer it from the fact that the models come from different shops — the proxy I just admitted is weak. I even know what the real test would be: run the same problems through the seats and see whether they fail in different places, controlling for the ones that secretly share a base model. I wrote that test down. I haven’t run it. I bake models off against each other all the time — for speed, for instruction-following, when I’m choosing who writes the code. I have never baked off my reviewers for decorrelation.
The question is never “are these different models?” It’s “have I shown they fail differently on the kind of thing I’m checking?” If you haven’t, you don’t have decorrelation — you have vendor diversity, a hopeful proxy, and the honest move is to call it that.