LLM Evals in 2026: A Practitioner's Field Guide
Last week I watched a team ship a support agent that passed 94% of their eval suite and then hallucinated a refund policy on day two of production. Their eval was measuring the wrong thing, well. This guide is what I wish someone had handed me the first time I built an LLM eval harness that actually caught regressions before customers did.
What "LLM evals" actually means in 2026
An LLM eval is a repeatable measurement of a model or agent's output quality on a fixed dataset, using deterministic checks, model-graded judges, or human review, run in CI or on a schedule so you can detect regressions before shipping. In 2026 the term covers four distinct things people conflate: offline benchmarks (fixed dataset, run pre-merge), online evals (production traces sampled and scored), guardrails (runtime checks that block bad outputs), and judge calibration (measuring whether your LLM-as-judge agrees with humans). If your team says "we have evals" and cannot tell you which of the four they mean, they do not have evals. They have a vibes-based test set.
The mental model I use with clients: evals are unit tests for probabilistic systems. They will not be green/red. They will be distributions. Your job is to make the distribution shift visible.
The eval framework landscape, honestly compared
I have used all four of these on real projects in the last 18 months. Here is how I actually pick between them.
| Framework | Best for | Self-host | LLM-judge built in | Trace-linked | Where it hurts |
|---|---|---|---|---|---|
| Braintrust | Teams that want a polished UI and fast iteration on prompts | No (SaaS) | Yes | Yes | Cost at scale, vendor lock on eval definitions |
| LangSmith | Teams already deep in LangChain/LangGraph | Limited (enterprise) | Yes | Yes, tightly | Framework gravity: hard to leave once your traces live there |
| Promptfoo | Config-as-code, CI-first, model comparison | Yes | Yes | Weak (bring your own tracing) | UI is basic, no real prod monitoring |
| Ragas | RAG-specific metrics (faithfulness, context precision) | Yes | Yes | No | Only useful if your problem is retrieval, judge metrics are noisy without calibration |
My default stack for a new client engagement: Promptfoo in CI + a lightweight custom dashboard on Postgres for online evals. I add Ragas only when the RAG faithfulness question is central. I recommend Braintrust when a non-engineering PM needs to run experiments without opening a PR. LangSmith is fine if you are already on LangChain and not planning to leave. I do not use it as a starting point in 2026 because the migration cost later is real.
The single biggest mistake I see: teams pick the framework before they have written 20 real eval cases by hand. The framework is 10% of the problem. The dataset is 90%.
The 12 failure modes I check for on every LLM system
This is the table I keep in my head, and now on paper. Each row is a failure I have shipped or watched someone else ship in production. Each has a specific check.
| # | Failure mode | What it looks like | How I catch it |
|---|---|---|---|
| 1 | Prompt drift | Small prompt edit tanks 3 unrelated cases | Snapshot eval on every prompt PR, diff scores |
| 2 | Hallucinated citations | Agent invents URL, doc ID, or policy | Regex check against known-good ID list + faithfulness judge |
| 3 | Tool-call malformation | JSON args missing required field | Schema validation on every tool call, fail loud |
| 4 | Silent tool failure | Tool returns empty, agent proceeds as if it worked | Assert non-empty tool return, log to trace |
| 5 | Context poisoning | Bad retrieval chunk flips the answer | Retrieval eval separate from generation eval |
| 6 | Judge bias | LLM judge always prefers longer answers | Calibrate judge against 50 human labels quarterly |
| 7 | Refusal creep | Model version upgrade adds unwanted refusals | "Should answer" set: track refusal rate over time |
| 8 | Loop / infinite planning | Agent replans same step forever | Max-steps guardrail + step-count metric |
| 9 | Cost blowup | One query burns 400k tokens | Per-trace token budget alert |
| 10 | Latency regression | p95 doubles after model swap | Latency in the eval report, not just quality |
| 11 | PII leakage | Model echoes user data into logs or downstream calls | Regex + entity detector on outputs |
| 12 | Eval set rot | Test cases become trivial after months of prompt tuning | Rotate 10% of eval set per quarter with real prod failures |
If you build a scorecard around these 12, you will catch roughly 80% of what actually breaks LLM systems in production. The remaining 20% is domain-specific, and that is where the interesting engineering happens.
The workflow I use to build an eval suite from scratch
Every engagement follows the same rough sequence. It takes about a week for a scoped agent, three weeks for something ambitious.
Step 1: Collect 50 real inputs, not synthetic ones. If the system is not live yet, sit with the people who will use it and ask them to type the messages they would actually send. Synthetic eval inputs generated by GPT are the reason so many production systems fail on real traffic. The distribution is wrong.
Step 2: Hand-label expected behavior for each input. Not expected output, expected behavior. "Should refuse", "should call search_docs with query about pricing", "should ask a clarifying question". Behavior labels survive prompt changes. Exact-output labels do not.
Step 3: Write deterministic checks first. Anything you can catch with a regex, a JSON schema, or a substring match, catch that way. Deterministic checks are free, fast, and never disagree with themselves. I usually get 30-40% of my checks deterministic.
Step 4: Add LLM-as-judge for the rest, then calibrate. Write your judge prompt. Run it on 50 cases. Have a human (usually me and the domain expert) label the same 50. Compute agreement. If your judge disagrees with humans more than 15% of the time, the judge prompt is broken, not the model. I wrote about this in more depth in my post on Hamel's method vs a lean in-house setup.
Step 5: Wire it into CI and a nightly cron. CI blocks PRs that regress the eval score by more than 2 points on any critical metric. The nightly job runs against a sample of yesterday's production traces so you catch drift the fixed set will not.
Step 6: Publish the numbers where the team sees them. A Slack post every morning with the score, a small table of what regressed, and a link to the diff. If the number lives in a dashboard nobody opens, you do not have evals. You have decoration.
Real numbers from a recent build
For a content generation agent I shipped this year, the eval investment looked like this. I am sharing the actual numbers because "evals are worth it" without numbers is a slogan.
- Time to write initial 60-case eval suite: 9 hours
- Time to calibrate the LLM judge to 91% human agreement: 4 hours
- Model + judge cost per full eval run: $0.42
- CI overhead added per PR: ~90 seconds
- Regressions caught before merge in first 8 weeks: 11
- Estimated cost of those 11 regressions reaching prod (rework + reputation): I do not know, but the two that involved wrong pricing claims would have been bad.
The eval suite paid for itself the first time it caught a prompt tweak that broke tool calling on 4 out of 60 cases. That took 90 seconds in CI. Debugging it in production would have taken hours and would have shipped wrong outputs to real users first.
Where LLM-as-judge quietly lies to you
This is the section I wish someone had written for me two years ago. LLM judges have systematic biases and if you do not measure them, your eval score is a lie you are telling yourself.
Position bias. In pairwise comparison ("is A or B better?"), most judges prefer whichever answer comes first. Fix: run every pair twice, swapped, and only count agreement.
Length bias. Judges prefer longer answers even when shorter is correct. Fix: include "concise" as an explicit criterion in the rubric, or normalize by length.
Self-preference. GPT-4 class judges prefer GPT-4 class outputs. Claude prefers Claude. If you use the same model family to generate and judge, you are grading your own homework. Fix: use a different family for the judge, or triangulate with two judges.
Rubric ambiguity. "Rate helpfulness 1-5" produces noise. "Score 1 if the answer cites the correct policy section by ID, 0 otherwise" produces signal. Push rubrics toward binary or ternary decisions with explicit criteria whenever you can.
I recalibrate judges every time I change the underlying model version. When Anthropic or OpenAI ships a new snapshot, judge behavior shifts. Skipping calibration and trusting the score is how teams end up with green dashboards and angry users.
What I would do if I were starting today
If you are a founder or head of engineering standing up an LLM system in 2026, this is the shortest path I know:
- Do not pick a framework yet. Write 50 real eval cases in a spreadsheet, by hand.
- Ship Promptfoo in CI on week one. It costs nothing, it works, and the config format is portable if you outgrow it.
- Track the 12 failure modes above as a checklist, not a philosophy. Tick them off one by one.
- Calibrate your judge quarterly against fresh human labels. Put it on the calendar.
- Rotate 10% of the eval set every quarter with real production failures. Otherwise the set rots and your scores lie.
- Do not spend more than 20% of your engineering budget on the eval framework itself. The dataset and the judge rubric are the leverage points. The tooling is a commodity.
The teams I see get this right treat evals the way good backend engineers treat observability: a first-class part of the system, built as the system is built, not bolted on after the first outage.
Closing
Evals in 2026 are not a checkbox. They are the difference between an LLM system you can operate with confidence and one you are constantly patching in response to angry users. If you are somewhere in that gap and want a second set of eyes on your eval strategy, or want help standing one up from scratch, I take a small number of these engagements each quarter: lazar-milicevic.com/#contact. More posts on production LLM work live on the blog if you want to keep reading.
Last updated: July 2026. Written by Lazar Milicevic, Senior Technical Engineer, 10+ years in AI automation, cloud architecture and B2B SaaS. Founder of BizFlowAI.
Frequently asked questions
What exactly is an LLM eval in 2026?
An LLM eval is a repeatable measurement of a model or agent's output quality on a fixed dataset, using deterministic checks, model-graded judges, or human review, run in CI or on a schedule so you can catch regressions before shipping. In practice the term covers four distinct things people conflate: offline benchmarks (fixed dataset, pre-merge), online evals (sampled production traces scored after the fact), guardrails (runtime checks that block bad outputs), and judge calibration (measuring whether your LLM-as-judge agrees with humans). If a team says 'we have evals' but cannot tell you which of the four they mean, they have a vibes-based test set, not evals. I treat evals as unit tests for probabilistic systems: the output is a distribution, and the job is to make distribution shifts visible.
Which LLM eval framework should I use: Braintrust, LangSmith, Promptfoo, or Ragas?
My default starting stack for a new project is Promptfoo in CI plus a lightweight custom dashboard on Postgres for online evals, because Promptfoo is config-as-code, self-hostable, and good for model comparison. I add Ragas only when RAG faithfulness and context precision are the central problem. I recommend Braintrust when a non-engineering PM needs a polished UI to run experiments without opening a PR, accepting the SaaS cost and some vendor lock-in. LangSmith is fine if you're already deep in LangChain/LangGraph, but I don't pick it fresh in 2026 because migration cost later is real. The bigger point: the framework is 10% of the problem and the dataset is 90%, so don't pick tooling before you've written 20 real eval cases by hand.
What are the most common failure modes in production LLM systems and how do I catch them?
The twelve failure modes I check on every LLM system are: prompt drift, hallucinated citations, tool-call malformation, silent tool failure, context poisoning, judge bias, refusal creep, infinite planning loops, cost blowups, latency regressions, PII leakage, and eval set rot. Each has a specific check: schema validation on tool calls, faithfulness judges plus regex against known-good IDs for citations, retrieval evals kept separate from generation evals, max-steps guardrails, per-trace token budgets, latency tracked in the eval report, and quarterly judge calibration against human labels. A 'should answer' set catches refusal creep after model upgrades, and rotating 10% of your eval set per quarter with real production failures prevents eval set rot. In my experience a scorecard around these twelve catches roughly 80% of what actually breaks LLM systems in production.
How do I build an LLM eval suite from scratch?
I follow the same four-step sequence on every engagement, and it takes about a week for a scoped agent or three weeks for something ambitious. Step 1: collect 50 real inputs from actual users, not synthetic ones generated by GPT, because synthetic distributions are the reason so many systems fail on real traffic. Step 2: hand-label expected behavior (like 'should refuse' or 'should call search_docs with a pricing query'), not expected output, because behavior labels survive prompt changes and exact-output labels don't. Step 3: write deterministic checks first using regex, JSON schema, or substring matches, since these are free, fast, and never disagree with themselves, and typically cover 30-40% of your checks. Step 4: add an LLM-as-judge for the rest and calibrate it against at least 50 human-labeled cases before trusting it.
Why did my LLM system pass 94% of its evals but still fail in production?
Almost always because the eval was measuring the wrong thing well, and I've seen this exact pattern many times, including a support agent that hallucinated a refund policy on day two of production. The most common root causes are synthetic eval inputs that don't match the real user distribution, exact-output labels instead of behavior labels so prompt tuning silently overfits, and eval set rot where cases become trivial after months of iteration. Teams also frequently confuse offline benchmarks with online production evals and guardrails, so they never notice regressions on live traffic. The fix is to collect 50 real user inputs, label behavior rather than output, add runtime guardrails and sampled online evals, and rotate 10% of your eval set each quarter with actual production failures.
Lazar Milićević
Senior Technical Engineer. I build AI automation, GenAI/LLM systems and cloud architecture — autonomous systems that run while you sleep. Founder of BizFlowAI.
Building something hard with AI or automation? I am open to talk.
Get in touch