Why 95% of AI Agent Pilots Never Ship

Cisco's number has been rattling around my head for weeks: 85% of enterprises are piloting AI agents, 5% have them in production. At VB Transform this week, Amazon's Bryan Silverthorn (Director of AGI Autonomy, formerly Adept AI) put a name on the gap. It isn't capability. It's reliability. That matches what I see every time I get pulled into a stalled agent project: the demo works, the pilot works, and then someone tries to run it 10,000 times unattended and the whole thing falls over.

I want to walk through the reliability engineering I actually use to move agents from "impressive on a Tuesday" to "runs at 3am on a Sunday without paging anyone." No benchmarks, no hype. The real stuff.

The 85/5 gap is a reliability tax, not a model problem

Silverthorn's point, translated into engineer-speak: a model that succeeds 90% of the time on a single step will succeed on a 10-step task about 35% of the time. Chain 20 steps and you're at 12%. That's not a benchmark issue. That's compounding failure, and no amount of GPT-6 or Claude 5 fixes it for you, because as the model gets smarter, product managers ask it to do longer chains.

The gap between pilot and production is almost entirely the work of driving per-step reliability from ~90% to ~99.5% and then designing the system so the remaining 0.5% degrades gracefully instead of catastrophically. In practice that means five things I do on every serious agent build:

1. Deterministic tools around a probabilistic core
2. Structured evals wired into CI, not spreadsheets
3. Retry and fallback policies that assume the model will misbehave
4. Guardrails at the tool boundary, not the prompt
5. Observability that lets you replay any failed run bit-for-bit

Everything below is how I do those five in production.

Deterministic tools around a probabilistic core

The single biggest reliability win: shrink the surface area where the LLM has creative freedom. The model should decide which tool to call and with what arguments. It should not be freeform-generating SQL, HTTP requests, or file paths.

Concretely, on the BizFlowAI ContentStudio pipeline I run every day, the agent that publishes posts has exactly seven tools. Each tool is a typed function with a JSON schema, input validation, and idempotency keys. The agent cannot "write to the database." It can call publish_draft(post_id, target_site) and that function does the writing.

@tool(schema=PublishDraftSchema)
def publish_draft(post_id: str, target_site: Literal["a", "b", "c"]) -> PublishResult:
    if not repo.draft_exists(post_id):
        return PublishResult(ok=False, reason="draft_not_found")
    if repo.already_published(post_id, target_site):
        return PublishResult(ok=True, reason="idempotent_hit")
    ...

Two things happen when you do this. First, hallucinated tool calls become type errors, not production incidents. Second, you can unit-test the tools without ever calling the model. My rule of thumb: if a tool cannot be safely called 100 times in a row by a drunk intern, it isn't ready for an agent.

Evals that block the deploy, not the pretty dashboard

Most teams I audit have "evals" that live in a Notion page. That is not an eval. An eval is a test that runs in CI, has a pass threshold, and blocks the merge when it fails.

Here is the setup I use, which is boring on purpose:

• Golden set: 30 to 200 real inputs from production traffic, hand-labeled with the correct final state (not the correct tokens, the correct outcome).
• Rubric evals for open-ended outputs, scored by a separate judge model at temperature 0, with the rubric versioned in git.
• Trajectory evals for multi-step agents: did it call the right tools in a valid order, or is it just accidentally landing on the right answer?
• Regression gate: new prompt or model change must not drop any category below its floor. A 2% overall improvement that tanks the "refund" category by 15% is a rollback.

I run this on every PR that touches a prompt, a tool schema, or the model version. It costs me maybe $3 to $8 per full run on the Claude and OpenAI APIs. Compared to shipping a broken agent, it's free.

One nuance most people miss: evaluate the trajectory, not just the answer. A GPT-class model will often stumble into the right final response through a sequence of steps that would break in a slightly different context. If you only grade the last message, you are training yourself to trust luck.

Retry and fallback policies that assume the model will misbehave

Every network engineer knows to retry on 503. LLM engineers keep forgetting. Here's the retry taxonomy I use, with the actual policies:

That last row is the one people skip and later regret. If the agent is confidently wrong, retrying with the same prompt gives you the same confident wrong answer at the same cost. Escalation is a first-class outcome, not a failure of the design.

I also set a hard step budget on every agent run. If it hasn't finished in N tool calls or M tokens, it stops and hands off. Unbounded agents are how a $12 task becomes a $340 incident.

Guardrails belong at the tool boundary

Prompt-level guardrails ("do not delete user data") are wishful thinking. They work maybe 95% of the time, which is a catastrophe if the 5% deletes user data.

Real guardrails are enforced by the code that runs the action, not the model that requests it. The pattern:

def delete_user(user_id: str, requested_by: AgentContext) -> Result:
    # Guardrail 1: agent identity
    if requested_by.role != "admin_agent":
        return Result.deny("insufficient_role")
    # Guardrail 2: rate limit per agent per hour
    if rate_limiter.exceeded(requested_by.agent_id, "delete_user"):
        return Result.deny("rate_limited")
    # Guardrail 3: soft delete only, 24h reversal window
    return soft_delete(user_id, reversible_until=now() + 24*hour)

Three cheap guardrails. None of them trust the prompt. The agent can hallucinate a delete call and it still cannot actually destroy anything irrecoverable.

On the Zendesk + AWS integration I built that hit first-ever SLA compliance, this pattern is why I could sleep. The Lambda functions that acted on ticket data had allowlists for actions and hard caps on batch sizes. If the routing logic ever went off the rails, the blast radius was one ticket, not a queue.

Observability that lets you replay any run bit-for-bit

If you cannot reproduce a failed agent run, you cannot fix it. This sounds obvious until you realize how many teams log "the agent errored" and nothing else.

The minimum I log for every agent run:

• The full input, including any retrieved context (RAG chunks with their IDs and scores)
• Every model call: model name and version, temperature, seed if available, system prompt hash, full messages, full response, token counts, latency
• Every tool call: name, arguments, result, duration
• The final state and outcome label

Stored in Postgres with a run_id and indexed by user, tenant, agent name, and outcome. Cost: a few cents per run in storage. Value: when someone says "the agent did something weird yesterday," I can pull the exact run in 20 seconds and replay it against a new prompt to see if the fix works.

I also sample maybe 2% of successful runs into the eval set weekly. Production traffic is the best source of edge cases you never imagined.

The number that actually matters: mean tokens to recovery

Benchmarks tell you how good your agent is when things go right. In production, what matters is how quickly it gets back on track when things go wrong.

I track a metric I call MTTR-tokens: on runs that eventually succeed, how many extra tokens (and dollars) did the agent burn recovering from its own mistakes? If that number is climbing week over week, your reliability is degrading even if success rate looks flat. If it's dropping, your retry and self-correction logic is genuinely working.

For a well-tuned production agent I'd expect:

• Success rate: 98%+ on the golden set, 95%+ on live traffic
• P95 latency: under 2x the P50, or your retry loop is unbounded
• MTTR-tokens: under 20% of a nominal successful run
• Human escalation rate: 1 to 5%, with a clear queue and SLA
• Cost per successful task: stable or trending down as you add caching

If you don't know these numbers for your agent, you don't have a production agent. You have a demo with users.

What I'd do if I were you and stuck at pilot

Blunt version, because the 85/5 gap is mostly self-inflicted:

1. Freeze new capabilities for two weeks. Do not add another tool, another workflow, another use case. You are not capability-bound, you are reliability-bound.
2. Build a 50-item golden eval set from real traffic. Label the correct outcomes. Wire it into CI. This is a one-week job for one engineer and it will change how the whole team makes decisions.
3. Audit your tools. For each one, ask: can I call this 100 times with random inputs and not corrupt anything? If no, add validation, idempotency, and rate limits before you touch the prompt again.
4. Move guardrails from prompts to code. Every "the LLM shouldn't do X" becomes "the tool refuses to do X."
5. Instrument for replay. If you can't replay yesterday's failure today, fix that before anything else.
6. Set a step budget and an escalation path. Unbounded loops and no human fallback are how pilots die.
7. Ship a boring version of one workflow end-to-end. One workflow in production beats ten in pilot every single time.

Silverthorn is right that better models won't close the gap alone. What closes it is treating agents like the distributed systems they are: unreliable components composed into a reliable whole through boring, disciplined engineering. Retries. Validation. Idempotency. Observability. The stuff nobody demos.

I have been building this way for years across content pipelines, serverless integrations, and RAG systems, and the same pattern shows up every time. The teams that ship are not the ones with the best prompts. They are the ones who accepted that the model is a component, not a product.

If you're stuck in the 85% and want a second set of eyes on why, or you want someone to help you build the reliability layer around your agents so they can actually ship, drop me a note at lazar-milicevic.com/#contact. More production notes on the blog if you'd rather read first.