Read most coverage of AI agents and you’d think the whole game is the model — which lab is ahead this week, how many tokens, which benchmark moved. It’s the part everyone can see, so it’s the part everyone argues about. But if you’ve actually built an agent that runs in production and that real people depend on, you learn something quickly: the model is the smallest part of a working system. What makes an agent reliable, useful and affordable is almost entirely the stuff around the model — the harness, the boundaries, and the discipline of using a right-sized model for each job. “It’s just LLMs and tokens” is how you end up with a demo that dazzles and a system that never ships.
The harness is the system
Start with the harness — the scaffolding the model runs inside. It’s the tools the agent can call, the memory it reads and writes, the retrieval that grounds it in real data, the orchestration that sequences steps, the guardrails that constrain it, and the evals that tell you whether any of it actually works. The model is one component plugged into that system; the system is where reliability lives.
Swap a better model into a bad harness and it’s still bad. Build a great harness around a modest model and it becomes genuinely useful. When I built an assistant for my own family, the model choices mattered far less than the plumbing around them — the deterministic data layer, the propose-then-act gate, the memory that let it recall commitments. That isn’t a quirk of a hobby project; it’s the general shape. The harness dominates the outcome, and the model is a part you can swap.
Boundaries: ground the facts, then generate
The single most important design decision in any agent is where you draw the line between deterministic and probabilistic. Language models are extraordinary at judgment, phrasing and pattern. They are unreliable at being exactly right about a specific fact. So the discipline is simple to state and hard to skip: ground the facts deterministically, and let the model reason and phrase.
A deterministic step pulls the real data — the balance, the date, the record — from a system that can’t hallucinate. The model only ever composes from those given facts; it never invents a number or a time. This one split does two things at once. It kills a whole class of hallucination, and it makes the agent testable, because the deterministic parts have right answers you can assert against. The corollary is knowing what to keep on the deterministic side of the line: anything correctness-critical — a calculation, a business rule, a transaction — should be a deterministic service the agent calls, not something you hope the model reproduces correctly this time.
Right-size the model — route by task, not by hype
Here is where “more tokens, bigger model” does the most damage. Not every step in an agent needs a frontier model. A great deal of the work — classify this message, extract these fields, format this output, translate this line — is handled perfectly by a small, fast, cheap model, often one you can run locally. The reasoning and planning steps might warrant something stronger. The genuinely hard, open-ended generation might justify the frontier. Matching each task to the right-sized model — by cost, latency, and the reliability the task actually needs — is one of the highest-leverage decisions in the whole system.
“Throw the biggest model at everything” is one of the most expensive mistakes in the field. It’s slower, it’s costlier, and it’s frequently no better — because the hard part was never the raw capability. It was the harness feeding the model the right context. A well-fed small model beats a poorly-fed large one more often than the token-count discourse would ever admit.
Evals and guardrails are first-class, not afterthoughts
Two things separate an agent you can trust from a demo you can’t, and both have to be built in from the start rather than bolted on later.
Evals are how you know whether a change helped or hurt. Without them you’re tuning by vibes — and agents are far too non-deterministic to tune by vibes. A change that looks better on three hand-tried examples can be worse across a hundred; only a real eval set tells you which. Guardrails are how you bound the blast radius for when the model is wrong, which it will be: input validation, output checks, rate limits, and a human-approval gate on anything with a real-world effect. In a serious agent these aren’t features you add once something goes wrong. They’re part of the frame you build inside from day one.
Why the framing matters
“Just LLMs and tokens” isn’t only imprecise — it leads directly to bad architecture. It makes teams over-index on the model and the model provider, skip the deterministic boundaries, reach for one giant model to do everything, and ship without evals. Then the demo works and production doesn’t, and the diagnosis comes back “we need a better model” — when the real problem was everything around the model. More capability rarely fixes a harness problem; it just makes the same mistakes more expensively.
The teams that ship reliable agents do the opposite. They treat the model as a swappable component and pour their engineering into the harness, the boundaries, and the routing. When a better model arrives, they slot it in and their system gets better for free — because the hard, durable work was never the weights.
The engine and the car
The model is the engine. The harness is the car, the road, and the driver. You don’t win a race by dropping a bigger engine into a car with no steering — you win with the whole system engineered to work together. And as the models keep leapfrogging each other week to week, that’s oddly reassuring: if the model is the swappable part, then the durable work — the part that’s actually yours — is the harness you build around it. Ground the facts, bound the actions, right-size the models, and measure everything. That’s the part “LLMs and tokens” never mentions, and it’s the part that decides whether your agent is a toy or a tool.
Part of the Agentic AI series
- Agents Eat the Stack Top-Down
- The Substrate War
- Buy, Build, or Orchestrate
- Service-as-Software
- The Frontier Labs Are in the Squeezed Middle of Their Own Stack
Companions: Who Advises the Disrupted? · Agentic AI Is Not Just LLMs and Tokens — you are here · A Product Playbook for Agentic AI · Why Agentic AI Transformation Is Hard · Give Your Coding Agents a Process · A Longer Leash.
Leave a Reply