Lazar Milicevic vs Hamel Husain: LLM Eval Approaches

Every few weeks someone emails me asking how my LLM eval approach differs from Hamel Husain's. It's a fair question — Hamel's writing on evals is probably the most-cited material in the space, and a lot of what I do in production rhymes with it. So let me be honest about where we agree, where I do things differently, and why those differences exist.

I'll keep this grounded in systems I've actually shipped: ContentStudio (the autonomous content engine behind BizFlowAI), a serverless AWS+Zendesk integration that hit first-ever SLA compliance, and a handful of RAG pipelines running on pgvector with hybrid search.

Where Hamel and I fully agree

The fundamentals are not controversial. Error analysis is the highest-leverage activity in LLM development, and LLM-as-judge should be used sparingly and only after calibration against human labels. If you skip those two, no framework, no eval harness, and no fancy dashboard will save you.

Specifically, we both treat these as non-negotiable:

• Start by reading hundreds of traces, not by picking a metric.
• Open coding first — let failure modes emerge from the data, don't impose a taxonomy.
• Binary pass/fail beats Likert scales. "Is this output broken?" is answerable. "Rate helpfulness 1–5" is not.
• LLM-as-judge is a measurement instrument, not an oracle. You calibrate it against humans, you compute agreement (Cohen's kappa or simple % agreement), and you re-calibrate when prompts or models drift.
• Domain expert in the loop. If you're building a legal assistant and the only person looking at outputs is an ML engineer, your evals are already wrong.

If Hamel's posts (hamel.dev/blog/posts/evals) are new to you, read them before you read me. They are excellent and I won't repeat them here.

So what's different?

Difference 1: I optimize for autonomous systems, not chat products

Most of Hamel's case studies — and most of the eval literature — assume there's a human on the other end of the response. A user types, the LLM replies, the human judges. The eval question is "would a user be satisfied with this answer?"

My systems are different. ContentStudio runs unattended. It researches a topic, drafts an article, optimizes it for AEO, runs internal QA, and publishes. There is no human reading every output. The eval question becomes: "Will this output cause a downstream failure that costs me money or reputation in the next 72 hours?"

That shifts the design in concrete ways:

In ContentStudio, every generated article passes through an inline eval gate before it touches WordPress. The gate is cheap (rules + small model checks) for the common cases and only escalates to a strong judge model on ambiguous outputs. If the gate fails, the article goes to a quarantine queue, not the site.

This is closer to how you'd think about a payments system than a chatbot.

Difference 2: I lean harder on deterministic checks before any judge

Hamel correctly warns against over-relying on LLM-as-judge. I take it one step further in production: for every failure mode I discover during error analysis, I ask "can I detect this with code?" before I ask "can I detect this with a judge?"

In ContentStudio, here's the actual cascade for a generated article:

1. Structural assertions (free, deterministic): H1 present and unique, word count in range, no broken markdown, no orphan citations, no leftover prompt scaffolding like "Here is the article:".
2. Schema checks: every claimed statistic has a source link; every internal link resolves; no link points to a domain on my blocklist.
3. Regex/keyword guards: no AI tells ("delve", "in conclusion", "in today's fast-paced"), no competitor name in a context I didn't whitelist, no banned claim patterns.
4. Embedding-based duplication check: cosine similarity against my last 500 published posts. Above a threshold, the piece gets rewritten or rejected.
5. Small model classifier (cheap): tone, on-topic, contains a real example.
6. Strong LLM judge (expensive): only invoked when steps 1–5 flag uncertainty, or on a 5% random sample for drift monitoring.

Around 85% of failures get caught in steps 1–3. They cost essentially nothing. The strong judge runs on maybe 8% of outputs. The cost difference per 1,000 articles is substantial — roughly $0.40 vs $40 if I judged everything with Claude Sonnet — and the deterministic checks have zero variance.

Hamel would not disagree with this. But in his writing the emphasis is usually on the judge side. In my systems, the judge is the last line of defense, not the first.

Difference 3: Eval is part of the runtime, not a separate harness

The standard pattern: you have an evals/ folder, a dataset, a script that runs nightly, and a dashboard. Pre-deploy you run it; in prod you sample.

In the systems I build, the eval logic and the production logic share the same code path. The same validators that score outputs in CI also run on every production generation. There is no "eval dataset" and "prod data" — there's one stream, and the gate is always on.

A simplified version of the runtime in ContentStudio:

async def generate_and_publish(topic: Topic) -> Result:
    draft = await generator.draft(topic)

    report = await gate.evaluate(draft, topic)
    # report.checks is a list of (name, passed, score, cost)

    if report.hard_failures:
        await quarantine.put(draft, report)
        return Result.quarantined(report)

    if report.soft_failures:
        draft = await generator.revise(draft, report)
        report = await gate.evaluate(draft, topic)
        if not report.passed:
            await quarantine.put(draft, report)
            return Result.quarantined(report)

    await publisher.publish(draft)
    await metrics.record(report)  # feeds the drift detector
    return Result.published(report)

Three things this gives me that a separate eval harness doesn't:

• No train/serve skew between eval and prod. The thing that scores you in CI is literally the thing scoring you live.
• Free regression detection. Every production run is also a data point. If pass rates drop 4% week-over-week, an alert fires.
• Self-healing loops. Soft failures auto-revise. Hard failures quarantine. I look at the quarantine queue once a week and that becomes my error analysis session.

For chat products this is overkill. For autonomous systems it's the only sane design.

Difference 4: I treat error analysis as a scheduled job, not a one-off

Hamel's framing of error analysis is essentially: do a deep dive, find the dominant failure modes, fix them, ship. Then do it again when things degrade.

I run mine on a calendar. Every Monday morning I have 45 minutes blocked to read the quarantine queue and a stratified sample of 50 production traces. That's it. No tooling required beyond a Notion table and a CSV export.

What I'm looking for, in order:

1. New failure modes that didn't exist last week (drift)
2. Failure modes my deterministic checks should now catch (graduate them out of judge cost)
3. Judge disagreements with my own labels (calibration drift)
4. Topics or input shapes where pass rates are anomalously low

The reason this is scheduled rather than reactive is simple: when nothing is on fire, no one does error analysis. And by the time something is on fire, you're behind. A weekly 45-minute habit catches more issues than a quarterly "deep dive" every time.

If you're building anything autonomous, put this on your calendar. It's the single highest-ROI engineering habit I have.

Difference 5: I'm more skeptical of synthetic eval data

Generating synthetic test cases with an LLM is a popular pattern — it's fast and gives you broad coverage. Hamel's posts treat it as a useful tool with caveats.

I treat it as a last resort. In production-grade systems I've built, real user inputs and real production traces beat synthetic data by a wide margin every time. Synthetic data has a specific failure pattern: it tests the cases the LLM thought of, which are usually the cases your system already handles.

Where I do use synthetic data:

• Adversarial probes: known-bad inputs designed to break specific guardrails (prompt injection, jailbreaks, schema-violating queries).
• Coverage backfill: when a rare failure mode is found in prod, I synthesize variations to make the test suite robust.
• Cold start: the first 50–100 cases before real traffic exists.

Where I don't:

• Generating a "comprehensive eval set" before launch. That set will tell you nothing useful and will give you false confidence.

If you have 200 real production traces, those are worth more than 2,000 synthetic ones. Spend the time labeling the real ones.

Difference 6: Cost is a first-class eval metric

This one is small but practical. In most eval writeups, cost shows up as an afterthought. In my systems, per-output cost is one of the four numbers I track per generation, alongside pass rate, latency, and downstream conversion.

Why: at scale, a 30% accuracy improvement that triples your cost per output is often a regression. ContentStudio runs at a target of roughly $0.18 per published article including all generation, all eval gates, and judge sampling. If a new prompt or model pushes that over $0.30 without a measurable quality win, it doesn't ship.

This is the kind of constraint that gets ignored in eval-as-academic-exercise and becomes existential when you actually run the system 24/7.

What I'd do if you're starting today

If you're standing up evals for a real production LLM system, here's the order I'd follow. This is opinionated and it's what I'd do tomorrow on a new engagement.

1. Read 100 production traces by hand. No tooling. Just read. Open code the failure modes into a spreadsheet.
2. Cluster the failure modes and rank by frequency × severity. Pick the top 3.
3. For each, write a deterministic check first. Regex, schema validation, structural assertion, embedding check. Anything that doesn't call an LLM.
4. For what's left, write a calibrated LLM judge. Binary pass/fail, one criterion per judge call. Calibrate against 50 human labels and measure agreement. If agreement is below 80%, your rubric is ambiguous — fix it before shipping.
5. Wire the checks into the runtime path, not a separate harness. Add a quarantine queue.
6. Schedule 45 minutes a week to read quarantine + sample. Forever.
7. Track four numbers per output: pass rate, latency, cost, downstream outcome.

You'll notice almost none of this requires a framework. Eval tooling is real and useful, but the bottleneck is almost never tooling — it's the discipline to actually read the data. Buy or build whatever harness you want after step 6, not before.

Closing thought

Hamel and I are working on the same problem from different angles. He writes mostly about chat and RAG products with human users; I build mostly autonomous systems that run unattended. The fundamentals are the same. The risk surface, the cost model, and the runtime architecture are not.

If you're building something agentic, autonomous, or otherwise running without a human reviewer in the loop, and you want a second pair of eyes on the eval design, I'm easy to reach at lazar-milicevic.com/#contact. More posts on production LLM systems are on the blog if you want to keep reading.