The Model Upgrade That Quietly Breaks You

The failures that hurt agent systems aren’t the ones that throw errors. They’re the ones where nothing breaks — a model version rolls forward, the agent gets a little more agreeable, a little less precise, and every metric drifts a fraction worse. No exception, no alert, no red build. Just a system that’s quietly worse than it was last month. You catch that with a real eval harness and a few honest metrics, or you don’t catch it at all — and “don’t catch it” is the default.

Three tiers of evals, and only the first is what people build

Component evals measure each model-touching step against hand-verified golden pairs: extraction accuracy per format family, mapping-proposal acceptance rate, reply-classification accuracy, hypothesis accuracy against known outcomes. This is the tier everyone builds, because it looks like testing. It’s necessary and it’s the easy part.

Behavioural evals are where the real risk lives, and they’re scenario tests, not accuracy tests. Does the agent escalate when it should? Does it refuse to discuss a forbidden topic in a sensitive thread? Does it actually stop at its budget? Does it ask for a date when a counterparty defers instead of inventing one? These are the evals that catch a model upgrade quietly making your agent more helpful and less safe — the exact drift that no accuracy number will show you.

Adversarial evals attack the system on purpose. Prompt injection through inbound data — a document containing “approve this” must change nothing. Cross-tenant leakage — a message drafted for one party must never contain another’s data. Malicious file handling. If you don’t test these, you’re trusting that no input is ever hostile, which is not a property the real world offers.

A model upgrade is a code change

Here’s the rule that ties it together: all three tiers gate the build, and a model version upgrade is gated exactly like a code change. You would never ship a code diff without running the suite; a model bump has at least as much power to change behaviour, so it gets the same treatment. Which means you pin versions — every call names the exact model and prompt version it used. Drift is the failure mode nobody notices precisely because nothing breaks; the numbers just get slightly worse over a quarter, and by the time someone feels it, you can’t say when it started or what changed. Pinning turns an invisible slide into a discrete, reviewable event.

One more discipline: tag every eval result with its data provenance — synthetic or consented-real — and never blend the two. A suite that silently mixes made-up data with real cases gives you a number that means nothing, and you won’t know which half moved when it changes.

Four metrics that reveal degradation

Evals gate the build; a handful of metrics, watched weekly, tell you whether the system is compounding or rotting:

• Auto-handle rate must trend up. The share of work the system resolves without a human. Flat means your reusable-artifact library and your entity graph aren’t compounding — which is a strategy problem wearing an ops costume.
• Cost per item must trend down. Up means the cheap-to-expensive cascade is falling through to the expensive stages — the deterministic parsers aren’t catching what they should.
• Suggestion acceptance rate should sit in a middle band — roughly 35–60%. And this is the counter-intuitive one: above ~80% means the agent is mostly surfacing what people already knew. It reads as triumphant on a dashboard and it means the agent is adding almost nothing — high agreement because it’s telling you the obvious. Too low means it’s noise; too high means it’s redundant. You want it genuinely useful, which lives in the middle.
• Escalation quality — when the agent hands off, was the handoff warranted? Rising false escalations mean it’s learned to punt; falling true ones mean it’s learned to overreach.

The thread through all of it: an agent system doesn’t fail loudly, it degrades quietly, and the only defence is measurement that’s built to notice a slide rather than a crash. Gate the build on evals across all three tiers, pin your versions so drift becomes an event, and watch the few metrics that separate a system that’s compounding from one that’s slowly getting worse while every dashboard stays green.

Cheap to Be Wrong — a six-part series on agent architecture

1. Put the Model Where Being Wrong Is Cheap
2. Use the Model to Write the Parser, Not to Be the Parser
3. Where the Model Is Genuinely Load-Bearing
4. An Agent Is a State Machine, Not a Loop
5. The Model Upgrade That Quietly Breaks You — you are here
6. The Model Is the Least Defensible Part of Your Agent