There’s an obvious objection to everything in this series, and it deserves a direct answer rather than a dodge. If the models keep getting better — more accurate, and now self-correcting, checking their own work before they answer — don’t the boundaries and the harness become scaffolding you can eventually remove? Won’t a good-enough model just be trusted with the decision, parse the data reliably, and make the discipline look like yesterday’s caution? It’s the right question to ask of any architecture doctrine. The answer is that model progress moves what sits above the line — not the line itself, because the line was never drawn by how good the model is.
What genuinely shifts
Plenty does change, and it’s worth being honest about it. A better model expands its own territory: documents it couldn’t reliably read become readable, intents it confused become distinguishable, the short list of genuinely load-bearing jobs grows. Cheaper, faster inference softens the raw cost argument for pushing work off the model — running a model per item stings less every year. And self-correction genuinely lowers the error rate on the model’s own turf, which means fewer review items and less obvious slop. All real, all welcome. If the case for this architecture rested on “the model isn’t good enough yet,” it would indeed weaken every quarter.
What doesn’t move, and why
But the case never rested on the model being weak. It rests on four things model progress doesn’t touch:
- Accountability. A consequential decision needs a name attached — a human who confirmed the rule, an owner who can be asked why. A more accurate model doesn’t supply accountability; it just makes the missing accountability easier to overlook.
- Reproducibility. A self-correcting model is still non-deterministic — run it twice and it can reason two different ways to two different answers. “It reconsidered and felt more confident” is not the same as “it gives the same answer every time,” which is what a decision path requires.
- Auditability. “Why did this become that” still has to be answerable after the fact, by someone who wasn’t in the room. Deterministic code answers it; a model — however clever — answers it with a story about its reasoning, which is not evidence.
- The economics of amortization. Doing the expensive understanding once and running code forever after beats doing it per-instance, no matter how cheap the instance gets — and it yields a compounding asset a competitor can’t rent.
None of these is a statement about capability. They’re statements about what a consequential, auditable decision is. That’s why the line — put the model where being wrong is cheap, code where being wrong is expensive or unauditable — holds at any model quality. The model’s side of the line gets bigger and better; the reasons for the line are unchanged.
Self-correction is not verification
The specific promise of a self-correcting model deserves its own answer, because it’s the most seductive. A model that checks its own work before answering is a real improvement — but it is still the model grading its own homework. Self-correction and independent verification are different things, and the gap between them is exactly where the expensive failures live: the confident, internally-consistent, wrong answer that passed its own check. The more capable and more persuasive the model, the more that self-certification can lull you. So the external, independent gate — a test that fails if the fix is reverted, a separate reviewer that doesn’t share the model’s assumptions — matters more as models self-correct, not less. (This is the same argument that applies to handing autonomous agents a longer leash: the check has to be independent of the thing being checked.)
The harness gets more necessary, not less
It’s tempting to think a smarter, more autonomous model needs less of a state-machine harness around it. The opposite is true. A self-correcting model burns more tokens and time inside a step, so hard budgets matter more. It produces more internal reasoning, so replay matters more to reconstruct what happened. It’s more capable, so its blast radius is larger and capability scoping matters more. And degradation gets sneakier — a self-correcting model can mask its own regressions by fixing some of them, so surface accuracy looks stable while behaviour drifts underneath, which is precisely why the behavioural and adversarial evals earn their keep. Better models don’t retire the harness; they raise the stakes on it.
The moat only widens
Here’s the part that improves fastest with model progress, and it’s the one people expect to get worse. As models get better, cheaper, and more commoditised, the model converges toward a utility everyone rents — which makes it less defensible, not more. A rising tide of capability lifts every competitor equally. What it doesn’t touch is the accumulated artifacts — the format libraries, the entity graph, the confirmed-rule library, the evidence chain — which are model-independent and compound over time. The better the shared model gets, the more the difference between two companies is the stuff the shared model can’t give either of them.
The line is drawn by consequence, not accuracy
So the honest summary: model progress is a tailwind for this architecture, not a threat to it. It expands what the model can safely do on its side of the line, and it makes the moat — the artifacts, produced with the model’s help and then owned by your code — more valuable, because the model itself is more commoditised. The one thing it does not do is move the line, because the line was never about the model being good enough. It’s about the fact that some decisions have to be accountable, reproducible, and auditable no matter who — or what — is making them. Build for that, and the next model, and the one after it, just make your system better without making it less trustworthy. Build against it, and no amount of model progress will save you — it’ll just make the same mistakes faster, and more convincingly.
Companion to the six-part series Cheap to Be Wrong — an architecture for agentic systems. See also Agentic AI Is Not Just LLMs and Tokens.
Leave a Reply