Why most agent pilots never reach production
The demo works. Nine months later it is still a demo. The gap is almost never the model — it is four things nobody scoped at the start.
Every organisation we talk to has an agent pilot. Roughly one in five has an agent in production. The difference is not model quality, budget, or engineering talent — those are broadly the same on both sides of the line. The difference is that the production teams scoped four things the pilot teams left until later.
1. Nobody defined what the agent is allowed to touch
A pilot runs against a sandbox with a friendly API key. Production means an agent that can write to your ERP, message your customers, or move money. The moment that becomes real, someone in security asks what it can reach — and if the answer is "whatever the API key allows", the project stops.
Scope tools explicitly, from the first week. An agent gets a named list of operations, each with its own credential and its own blast radius. Read operations are cheap. Write operations get a budget. Irreversible operations get a human.
2. There is no evaluation set, so nobody can say whether it works
Ask a pilot team how well their agent performs and you get anecdotes. Ask a production team and you get a number, because they built a scored task set before they built the agent.
The set does not need to be large. Thirty real cases, including the awkward ones, chosen by the people who do the work today, beats a thousand synthetic examples. What matters is that it is fixed, versioned, and run on every change — so when a model provider ships an update, a regression shows up as a number rather than an incident.
Without an evaluation set, "is it good enough?" has no defensible answer — which is exactly why the decision keeps getting deferred.
3. Failure was never designed, only discovered
Ordinary software fails loudly. Agents fail plausibly. They produce a confident, well-formatted, entirely wrong result, and unless someone is checking, it flows downstream looking like every correct result before it.
- A step budget, a time budget and a spend budget, each with a hard stop
- Defined behaviour when a tool errors — retry, escalate, or abandon
- A full run trace: the plan, every tool call, every retry, the cost
- A human handoff path that someone actually monitors
4. The pilot was scoped to impress, not to finish
Pilots get chosen for how well they demo. Production candidates get chosen for how narrow they are. The agent that triages one invoice type end to end ships; the agent that handles "finance operations" does not.
Pick the workflow where the volume is boring and the rules are known. That is where the payback is, and it is also where failure is survivable while you are still learning what the thing does under load.
If your pilot has been running for more than a quarter, the question is not whether the model is good enough. It is which of these four you skipped.
