The Enterprise AI Evaluation Gap Is Widening

Half of enterprises have shipped an AI agent or LLM feature that passed internal evaluations and still caused a customer-facing failure. One in four saw it happen more than once. That comes from a June 2026 VB Pulse survey of 157 enterprise respondents at companies with 100+ employees, and it matches almost exactly what I see when I get pulled into agent projects that are "almost ready."

The pattern is consistent. Teams are giving agents more autonomy (more tools, longer horizons, fewer confirmations) at the exact moment their trust in automated evals is collapsing. Nobody quite says it out loud, but the internal test suite has become a ritual, not a verdict. This post is about the gap between "eval green" and "production safe," and the specific patterns I use to close it before turning autonomy up.

Why passing evals no longer means safe to ship

The short version: most eval suites test the model, not the system. They score a fixed set of prompts against a fixed set of expected outputs, usually with an LLM-as-judge that was calibrated once and never re-checked. Meanwhile the agent that actually reaches production has tool calls, retrieval, memory, retries, a router, timeouts, and a user who says things no test writer imagined.

Here is what I check first when a team tells me their evals pass but production keeps breaking:

• Eval set is static and stale. The failure modes from last quarter are covered. This quarter's are not.
• Judge is uncalibrated. No one has computed judge-vs-human agreement on a labeled sample in months. The judge and the agent often share a model family, which quietly inflates scores.
• Coverage is on prompts, not on trajectories. Single-turn Q&A is measured. Tool sequences, partial failures, and multi-turn recovery are not.
• No distinction between capability and safety. The suite reports one number. A regression on refusal behavior looks the same as a regression on tone.
• Production traffic never flows back into evals. The "eval set" and "the things users actually type" are two disjoint distributions.

If four out of five are true, and they usually are, "we passed evals" is roughly as informative as "the unit tests are green" on a system with no integration tests.

The three-layer eval stack I actually ship

I stopped trying to build one heroic eval suite. In production I run three layers, each answering a different question. They are cheap to keep separate and expensive to entangle.

The unit layer is where LLM-as-judge lives, and only after I have calibrated the judge against 100-200 human-labeled examples. I want judge-vs-human agreement above 0.8 Cohen's kappa on the labels that matter (correctness, harmful, off-policy). Below that the judge is a random number generator with a confident tone.

The trajectory layer is where most teams under-invest. I run scripted personas against a staging agent with real tools (or high-fidelity mocks), and I score the whole trace: did it call the right tools, in a sensible order, did it recover when a tool returned an error, did it stop when it should have stopped. A single "final answer correct" score hides the interesting failures.

The production layer is where the evaluation gap actually closes, and it is what the rest of this post is about.

Tracing that lets you argue about a failure two days later

Every agent I put into production writes a structured trace per session. Not logs, traces. The difference matters: a trace is a single object with a stable ID that captures the entire causal chain, and it is the artifact I open when a customer complains on Thursday about something that happened on Tuesday.

The minimum useful schema, in the shape I actually store in Postgres (with pgvector on the message content for later similarity search):

{
  "trace_id": "uuid",
  "session_id": "uuid",
  "user_id": "hashed",
  "started_at": "ts",
  "agent_version": "git-sha",
  "prompt_version": "hash",
  "model": "claude-sonnet-4.x",
  "tool_calls": [
    {"name": "search_kb", "args": {...}, "latency_ms": 812, "status": "ok"},
    {"name": "create_ticket", "args": {...}, "latency_ms": 240, "status": "ok"}
  ],
  "messages": [{"role": "user", "content": "..."}, ...],
  "final_state": {"answered": true, "escalated": false, "tools_used": 2},
  "monitors": {"pii_leak": 0, "off_policy": 0, "self_contradiction": 0}
}

Two things I insist on:

1. Version everything the agent depends on. Model name, prompt hash, tool schema hash, retriever index version. When behavior changes, you need to know whether the model, the prompt, or the corpus moved.
2. Store traces long enough to matter. Ninety days minimum. Most incident postmortems I run reach back at least three weeks.

With this in place, "we cannot reproduce it" stops being a valid excuse. You replay the trace against the current agent and see what changes.

Online monitors: the evals that run on real traffic

Offline evals ask "would this have passed?" Online monitors ask "is this passing right now?" I run a small, cheap set on every production trace. They are not the same checks as the offline suite. They are designed to be fast, sampled where possible, and tuned for alerting rather than scoring.

The set I keep coming back to:

• Answer-grounding check. For any response that cites retrieved context, a lightweight NLI-style check on a 10-20% sample. If grounding drops, the retriever probably changed under me.
• Tool-error rate per tool. Broken out by tool. A slow creep from 2% to 6% on one tool is the first sign of an upstream API drift.
• Policy violations. A small classifier plus regex for things I truly cannot ship: PII in outputs, refusal-bypass patterns, disallowed topics. Cheap, fast, blocking.
• Trajectory shape. Distribution of tool-call counts, session length, escalation rate. Sudden shifts here almost always precede a user-visible incident.
• Self-consistency spot checks. For a small percentage of sessions, run the same input through the agent twice with different temperature seeds and compare. High disagreement on user-facing outputs is a leading indicator of regressions.

All of this streams into the same store as the traces, so when an alert fires I can pivot from "grounding dropped 8% in the last hour" to the exact 40 traces that triggered it.

Canarying an agent is not canarying a service

Standard progressive delivery, 1% → 5% → 25% → 100% by traffic, is the right shape. It is the wrong dimensions. With a stateless service, latency and error rate tell you almost everything. With an agent, the interesting failures do not show up in HTTP status codes.

What I actually gate on when promoting an agent version:

1. Task-completion rate on live traffic, compared to the previous version, on matched cohorts. If it drops more than 2 percentage points, roll back.
2. Escalation and abandonment rate. Users quietly giving up is the failure mode that never triggers a 500.
3. Tool-call cost per resolved session. A "smarter" agent that resolves the same tasks with 40% more tool calls is a regression, even if the answers are better.
4. Monitor-flag rate. Grounding, policy, self-consistency. Any of these worsening blocks promotion.
5. A human-review sample. For the first hour at each traffic step, a human reviewer looks at 30-50 sampled traces. Cheap, and it catches the qualitative stuff no metric will.

I keep the previous version warm for at least 24 hours after full rollout. Rollbacks on agents are more common than on regular services because the failure surface is larger and the discovery lag is longer.

Turning autonomy up without going blind

The failures in that VB Pulse number are, in my experience, almost always failures of autonomy calibration, not model capability. The agent was allowed to take an action it should have confirmed, or was given a tool that had a sharper edge than the eval suite tested.

I use a simple ladder when a team wants to give an agent more freedom. Every step up requires evidence from the layer below.

The autonomy ladder

• L0. Read-only. The agent can answer, cite, summarize. No side effects. Ship on unit evals + basic monitors.
• L1. Draft-and-confirm. The agent proposes writes (email, ticket update, CRM change) and a human clicks send. Promote when trajectory evals hit target and grounding monitor is stable for two weeks.
• L2. Confirmed autonomy on reversible actions. The agent executes reversible actions on its own, with a hard cap per session (e.g. max 3 writes) and a full audit trail. Promote when L1 shows human confirm rate above 95% for a month.
• L3. Autonomy on irreversible actions with guardrails. Payments, external emails, deletions. Requires per-action policy checks, spend caps, and a shadow reviewer sampling at least 5% of actions for the first month.
• L4. Full autonomy in scope. Rare. Only when L3 has run clean for a defined period on a defined action set.

Most projects I see try to jump from L0 to L3 in one sprint because the demo looked good. That is where the "passed internal evals, broke in production" number comes from.

What I would do this quarter if I owned an agent in production

If I inherited an enterprise agent tomorrow and had one quarter to close the eval gap, in order:

1. Week 1. Stand up structured tracing with versioned prompts and tool schemas. Nothing else matters until you can reproduce a failure.
2. Week 2. Label 200 real production traces by hand across correctness, policy, and trajectory quality. This becomes the seed of the real eval set and the calibration set for the judge.
3. Week 3. Compute judge-vs-human agreement. If it is under 0.7, replace the judge or the rubric. Do not skip this and pretend.
4. Week 4. Ship four online monitors: grounding, tool-error rate, policy, trajectory shape. Wire them to the same alerting stack as the rest of the platform. Page a human, not a channel nobody watches.
5. Weeks 5-6. Add trajectory evals with 20-40 scripted personas covering the tool graph, including partial-failure paths. Run nightly.
6. Weeks 7-8. Introduce the autonomy ladder explicitly. Document which actions are at which level, and what evidence is required to move up. Get the business owner to sign it.
7. Ongoing. Every production incident adds at least one case to unit evals and one persona to trajectory evals. The eval set grows with reality, not with imagination.

None of this is exotic. It is the same discipline good teams apply to any production system, adapted to a component whose failure modes are probabilistic and semantic instead of deterministic and structural. The reason it is rare in agent projects is that agents got shipped by ML teams who never operated services, or by product teams who never operated models. The gap is organizational as much as technical.

Closing

The evaluation gap is not going to close with a better benchmark. It closes when a team accepts that an agent is a distributed system whose behavior is defined by prompts, tools, retrieval, and traffic together, and instruments it accordingly. The teams that do this ship agents that keep working. The ones that do not end up in that 50% statistic, twice.

If you are staring at an agent that passes your evals and keeps embarrassing you in production, or you are about to turn autonomy up and want a second pair of eyes on the tracing, monitors, and canary plan, get in touch at lazar-milicevic.com/#contact. Or read more of what I have written about production LLM systems on the blog.