The review queue is the product

I am one person, and I am responsible for the AI inside more production systems than one person should be able to watch. Call scoring that listens to sales calls and decides which ones a coach should hear. Lead enrichment that researches companies and fills in what the CRM is missing. Planning statements drafted against local-authority policy. Articles generated and published. Site heroes generated and shipped. There is no rota and no second pair of eyes on most of it — just me, and a set of systems that each produce more in a day than I could read in a week. The honest question isn’t whether that’s impressive. It’s how it’s possible at all without the whole thing quietly filling up with mistakes I never see.

The answer is duller than automation and more important than any single model: where a system has earned it, I almost never look at its output — I look at what its review layer decides to surface. Each layer does a first pass of its own — scoring, checking, gating — and the ones I trust surface only the residue, the part the machine couldn’t resolve. Not all of them are there yet; some are still only measuring what they would have done. But where it works I can’t eyeball everything and don’t have to: I look at the small pile that’s left after the machine has reviewed the rest. The review isn’t a thing I do at the end. It’s a layer built into each system, and how far along that layer is decides how much still reaches me.

One move, many shapes

It’s really one move wearing different clothes in each product: automate the review so a human only ever sees what the machine couldn’t settle. The clearest case has no judgement in it at all. When the site generator makes a hero image, I don’t look at the image — an inspector scores it, and the pipeline ships the ones that clear the bar, tries to better the ones close to it, and keeps the best of several when it has to retry. That loop is live, and it’s the purest version of the move: the machine reviews its own output and self-heals on the failures, so I’m removed even from the rejects. I have published images I never saw, and that’s the system working as designed, not a corner cut.

Call scoring is the same move pointed at my attention rather than at quality. Calls get listened to and scored as they come in, but the system doesn’t hand me every one — a gate decides which are worth surfacing for coaching, with a daily cap so one bad morning doesn’t bury me, and a deterministic band on each so I can triage at a glance. Its job isn’t to do the coaching; it’s to make sure what reaches a human is the handful that earns the human, not the whole day’s traffic.

The planning statements are the same move again, with the dial turned up where the stakes are. A statement is drafted, and before it’s trusted a deterministic constraint-checker reads the planning context and computes how risky this one is — a flat extension on an unconstrained plot is not a basement dig under a Grade-I listing in a flood zone. Low risk gets a cheap model’s review; the expensive model is spent only when the constraints push the risk to red. And when I run that pipeline in its enforcement setting — which is a flag, off by default in the code, that I turn on in production — a failed high-risk verdict doesn’t ship: the system makes one repair attempt and then holds the statement for me. Off by default, on where I run it, and when it fires it holds rather than guesses. That nuance matters, and I’ll come back to it.

The same move guards the work I trust least, which is my own. The gate that lets a change into the systems running all this won’t pass while a reviewer has an open finding, won’t let the thing that built a change certify its own work, and won’t accept a claim of “it passes” without an executable check behind it. That gate is live and it is strict, because the cost of a bad merge into the review layer is every system downstream of it.

Not every shape is load-bearing yet, and this is where I have to be careful not to flatter myself. The lead-enrichment pipeline has a confidence floor built into it — the design is that a confident enrichment applies on its own and a doubtful one routes to a review queue. But I’m not running it that way. Today it’s in shadow: it routes everything to review while I watch how it would have scored, because I haven’t yet earned the trust to let it auto-apply anything. So it removes no review from my plate at all right now — it’s a measurement, not a saving. That’s deliberate, and it’s the same discipline as knowing when not to enforce: I’d rather run it dark and watch than flip it live and discover in production that the floor was set in the wrong place.

The cheapest reviewer is a machine that can’t lie to itself

The mechanism I lean on hardest is the one that does something an LLM grading an LLM can’t. When the content system generates an article, it pulls out the factual claims and checks each one — but not by asking a model whether it looks right. It checks the claim against a model-independent lookup: an exact query into the planning policy the claim is supposed to rest on. A language model reviewing another language model is attestation — a confident opinion, and confident opinions are exactly what these models produce when they’re wrong. The policy lookup is different in kind: it’s a fact the model didn’t author and can’t talk its way around. It catches the false-greens — claims that read perfectly and cite a policy that doesn’t say what they claim — that a judge model nodding along would wave through. I’ve written before that the verification that counts is the kind a model can’t author; this is that principle compiled into a job that runs so I’m not the one hand-checking every claim in them.

I have to name where that one actually stands, because an essay about honest verification that overstated its own keystone would be the exact failure it’s warning about. The oracle is on in my production setup — but it defaults off in the code, and I’m the one who turns it on. And it’s deliberately timid: when it can’t reach the ground truth it’s meant to check against, it doesn’t block and it doesn’t wave the claim through — it downgrades a “verified” claim back to “uncertain” and leaves it for a human. On in prod, off by default, conservative when it can’t see. That’s the honest shape of it, and the timidity is the point: a verifier that fails toward “look at this” is doing its job; one that fails toward “looks fine” is the thing I’m afraid of. The statement cascade I described earlier has exactly the same shape — on where I run it, off in the shipped default, holding rather than shipping when a high-risk statement fails its review — and I treat the two of them as the same kind of careful, not as finished guarantees.

You scale the review, not the model

Put those together and the operator economics come into focus, and they’re not the economics most AI writing talks about. The conversation is almost always about the model — which one, how big, how clever. But a better model doesn’t reduce the thing that actually bounds me. A cleverer model still produces output that needs checking; if anything it produces more of it, faster, and more plausibly wrong. What bounds a solo operator isn’t the quality of the generation — it’s the quantity of human review the generation demands. So the leverage isn’t a smarter model. It’s a review layer good enough that the residue reaching me is small, and trustworthy enough that I can clear it. You don’t scale by making the model better. You scale by making the review so good that less of the work needs you — and by being honest about how much still does. That’s the part that lets one person run what would otherwise need a team, and it’s the part nobody prints on a model card.

The second thing is quieter. This review layer is what makes running this much alone possible — it is not what makes it safe. “The review queue is the product” is as much a confession as a boast: it’s the only way I can carry this many systems at once, and on the days the queue is long it’s exactly as precarious as that sounds. I don’t want to dress the necessity up as a strategy I picked from a position of comfort. It holds, for now, because I’ve been careful about what I let off the leash — and careful, mostly, about admitting what I haven’t.

The question is never how good the model is. It’s how much of its output still has to reach you — and what your honest plan is to make that less without quietly making it look like less. Build the review until the residue is small and trustworthy, name the rung each piece is really on, and the queue that’s left is the product: it’s the thing that decides whether one person can run any of this at all.