You don’t want a smart AI. You want a boring one.
The goal was never a smarter agent. It’s a smaller job for the agent — one it can’t get wrong.
The pitch you’re being sold
An autonomous “AI employee” that reasons, plans, and adapts. When it underperforms, the advice is everywhere and mostly sensible: fix your prompt, add memory, connect more tools. What that advice almost never comes with is a number — a scored before-and-after.
We wanted numbers. So we built benchmarks and measured.
Why you don’t want the smart one
The moment an agent makes its own plans and its own judgment calls is the moment it can silently get a booking, a member, a payment wrong — and an agent rarely fails with an error you can see. It fails by succeeding. In a running operation, reasoning and adapting aren’t the feature, they’re the failure surface.
No business actually wants an improvising employee at the till. They want the work done the same way, every time, with a human called when something doesn’t fit.
To be fair to the smart agents: open-ended autonomy is genuinely useful — for exploration, for one-offs, for work you’ll review anyway. Just not for running your operation while you sleep.
What good engineering does instead
Decompose. The pipeline carries the structure — what’s known, what’s valid, what actions exist, when work is finished — and the model fills narrow, bounded blanks. No open-ended planning, no ambiguous judgment, because the pipeline removed the need for either. Boring describes the job, not the model: use as capable a model as you like — what it never owns is the state, the rules, or the final say.
Concretely: “AI, sort out the timetable clash” is a job for a smart agent, and it will eventually burn you. Our version is a decomposed pipeline: parse each line → match against live records → check for collisions → apply, or decline with a reason. There is no AI in it. Decomposed far enough, this workflow left nothing for a model to do — every step is deterministic, and every action is checked before it’s real.
We measured it
Two internal benchmarks, same construction: simulated customers with scripted intent — incomplete answers, corrections mid-conversation, items requested then retracted — then scored on the final result, field by field.
Both are research benchmarks: small by design, run while we’re still choosing the setup — before anything reaches deployment. And because the customers are simulated, the corpus scales whenever a decision needs more evidence. These two are here to show the shape of the effect.
Benchmark 1: multi-turn membership enquiries at a sports club. One model (Claude Haiku 4.5) — 15 conversations, 5 repeated runs. The only variable: how much the pipeline owns.
| what the pipeline owns | accuracy (5 runs) |
|---|---|
| Nothing — the model carries everything | 68–71% |
| The memory — the model reads it instead of remembering it | 90–92% |
| Memory, edits, validation, and which actions are even possible | 95–97% |
71% to 97% without the model getting one point smarter. When the pipeline defines the legal actions each turn, whole error categories become impossible to express — you can’t prompt your way out of a failure class.
Benchmark 2: quotation requests against a product catalogue. 42 conversations, 3 runs per configuration, two harness designs — one where the model re-reads everything and rewrites its draft, one where the pipeline owns the draft and the model proposes one bounded edit at a time.
| re-reads and rewrites | bounded edits, pipeline-owned draft | |
|---|---|---|
| Claude Haiku 4.5 | Pass — 0 errors ×3 runs | Fail — dropped a line on the longest conversation |
| DeepSeek V4 Flash | Fail — 20 of 42 never converged | Pass — 0 errors ×3 runs; 126 of 126 hand-audits exact |
Notice how the scores flip for DeepSeek V4 Flash — the bigger, more advanced model of the two? In one harness it failed half its conversations, unable to let go of items the customer had retracted; in the other, a perfect hand-audited score.
Reliability is not a property of the model alone. It’s a property of the setup: the model–harness pairing. This is something you configure, test, and measure. There is no “smartest” model to buy your way out with.
The dependency with a foreign-policy clause
There’s another reason to want the model’s job small, and it has nothing to do with accuracy. In June this year, a US export-control directive suspended all access to Fable 5 and Mythos 5 — for foreign nationals on paper, for everyone in practice, since nationality can’t be verified at an API. Access was restored nineteen days later. If your operation had been running on those models, you now know exactly what your contingency plan was worth for those nineteen days. Read June as a rehearsal, not a one-off: restricting frontier models is now a card up the sleeve of every government with a frontier lab in its jurisdiction. If your business depends on any frontier model, you’ve inherited that operational risk.
A decomposed pipeline hedges this twice over. First, a small, scored job rarely needs a frontier model at all. Second, the setup is measurable — if a model you depend on disappears tomorrow, moving off it is an engineering exercise: re-pair, re-run the benchmark, verify, ship.
Ask for the number
We began with the missing number, so let’s end there too. Nothing in our pipelines ships on a feeling: the benchmark re-runs, the score decides. Hold anyone selling you an agent to the same bar — ask how it was scored, on how many conversations, and what happened on the re-run. If the answer is a demo, you have your answer.
You don’t want an AI that’s clever. You want one that can’t surprise you.