LLM-as-a-Judge: Build, Calibrate, Trust It

The first LLM judge I shipped told me my content pipeline was producing 4.6/5 work. Then I sat down with three reviewers, blind-rated 50 of the same outputs, and the human median came back at 3.0. The judge wasn't broken. It was uncalibrated, sycophantic, and reading its own family's handwriting. That gap, 4.6 vs 3.0, is the thing nobody warns you about when they tell you to "just use an LLM to evaluate your LLM."

Here is the recipe I now use to build a judge I'd actually trust to gate a deploy.

Why naive LLM judges drift to 4.6/5

A direct-scoring judge with a prompt like "Rate this answer 1-5 for helpfulness" will almost always over-score. I've measured this on three different production systems and the pattern is consistent: a GPT-4-class judge run zero-shot lands 1.2-1.8 points higher than the human median on a 5-point scale.

Three reasons, in order of damage:

1. Sycophancy and positivity bias. RLHF models are trained to be agreeable. When you ask "is this good?", the prior leans yes.
2. Family bias. A GPT-class judge rates GPT outputs higher than Claude outputs on identical prompts, and vice versa. I've reproduced this on a 200-sample set with a delta of roughly 0.4-0.6 points.
3. Granularity collapse. On a 1-5 Likert, judges cluster on 4 and 5. You get almost no signal because 80% of scores are in a 1-point band.

If you don't fix these, your "eval" is a vanity metric. It will move when you change the prompt and not when you change the system, which is the exact opposite of what you want.

Step 1: Build a 150-sample human-labeled gold set

Before you write a single judge prompt, build the ground truth. This is the unglamorous step that determines whether the entire eval system is worth anything.

What I do, concretely:

• Sample 150 real production inputs. Not synthetic, not curated, not the easy ones. Stratify across the categories that matter (intent type, input length, hard vs easy, edge cases). 50 is too few to detect a 5-point Cohen's kappa shift, 500 is overkill for the iteration speed you need.
• Define a rubric with 3-5 atomic criteria. Not "is this good." Examples that work: factual accuracy against source, follows the user's constraints, no hallucinated entities, answers the actual question asked. Each criterion gets its own score.
• Three human raters per sample, blind. Pay them. Use Cohen's kappa to measure inter-rater agreement. If kappa < 0.6 between humans, your rubric is too vague. Fix the rubric, not the raters.
• Resolve disagreements by discussion, not majority vote. The cases humans disagree on are the cases that teach you what your rubric is missing.

This took me about 14 hours of focused work the first time, spread over two days. It's the single highest-leverage thing in the entire eval stack. Skip it and you're calibrating against vibes.

Step 2: Prefer pairwise over direct scoring

Once you have the gold set, the next decision is whether your judge scores one output at a time (direct) or compares two outputs (pairwise). I default to pairwise now, and only fall back to direct when I genuinely need an absolute number for an SLA.

Why pairwise wins in practice:

• Higher agreement with humans. On my content pipeline I measured pairwise judges at ~83% agreement with the human-resolved label, vs ~64% for direct scoring on the same gold set.
• Position bias is fixable. Judges prefer the first option ~55-60% of the time. You fix it by running each comparison twice with flipped order and only counting the result when both runs agree. Disagreement becomes "tie", which is itself useful signal.
• It maps to the decision you actually make. "Is the new prompt better than the old prompt?" is a pairwise question. Why fight it.

Here's the shape of the pairwise call I use:

def pairwise_judge(question, answer_a, answer_b, rubric):
    prompt = f"""You are evaluating two answers to the same question.
Apply this rubric strictly: {rubric}

Question: {question}
Answer A: {answer_a}
Answer B: {answer_b}

Think step by step about each rubric criterion.
Then output JSON: {{"reasoning": "...", "winner": "A"|"B"|"tie"}}"""
    
    forward = call_judge(prompt)
    # swap A and B
    reverse = call_judge(prompt_with_swapped(question, answer_b, answer_a, rubric))
    
    if forward.winner == flip(reverse.winner):
        return forward.winner
    return "tie"  # disagreement under position flip

The position-flip check alone moved my judge from 71% to 83% human agreement. It doubles your inference cost, which is fine because judges run on a sample, not on every production call.

Step 3: Calibrate with the gold set

Now you have a judge prompt and a gold set. Calibration is the loop that closes the gap between what the judge says and what humans said.

My calibration loop, in order:

1. Run the judge on all 150 samples. Record the score and the chain-of-thought reasoning.
2. Compute confusion against the human label. For pairwise: confusion matrix of {A, B, tie} predicted vs actual. For direct: mean absolute error and Spearman correlation.
3. Read every disagreement. Not just the metrics. Open the raw outputs. This is where the rubric gaps reveal themselves. I usually find 2-4 systematic failure modes in the first pass: "judge counts a partial answer as correct", "judge ignores citations", "judge rewards verbosity".
4. Rewrite the rubric to name the failure modes. Add explicit instructions like "An answer that lists three options without picking one does NOT satisfy 'answers the question'." This is where prompt engineering actually matters.
5. Add 3-5 worked examples to the judge prompt. Few-shot examples taken from your gold set disagreements. Show the judge a case where it would have said A, and explain why the right answer was tie.
6. Re-run, re-measure, repeat until agreement plateaus.

My target is >80% agreement with the resolved human label and Cohen's kappa > 0.65. Below that, the judge isn't a judge, it's a noise generator with API costs.

The number that matters most after calibration is not the agreement rate itself. It's the agreement rate on the disagreements that matter, meaning the borderline cases where humans themselves split 2-1. If your judge nails the easy 80% and is random on the contested 20%, your eval will make terrible deploy decisions exactly when the stakes are highest.

Step 4: Use a different model family than the one you're judging

This is small but mandatory. If your production system uses GPT-4o, judge with Claude. If it uses Claude, judge with GPT-4o or Gemini. The family bias I mentioned earlier is real and one-directional: a model is too kind to its own outputs.

I also run a two-judge consensus for any score that gates a deploy. Two different families, both must agree, and a disagreement gets a human in the loop. On the content pipeline, this catches the 5-8% of cases where one model has a weird blind spot the other doesn't. Cost is roughly $0.02-0.04 per judged sample with current pricing for mid-tier models, and you're judging a sample of traffic, not all of it.

Step 5: Wire the judge into CI, not just dashboards

A judge that lives in a Jupyter notebook is a hobby. A judge that fails a PR is infrastructure. My current setup:

• Nightly job runs the judge on a fixed 500-sample regression set against the last shipped prompt and the candidate prompt. If the candidate loses by more than a small margin (set per system, mine is 3 percentage points in pairwise win rate), the PR is blocked.
• Online sampling runs the judge on 2-5% of live traffic and writes scores to a time-series store. A sustained dip triggers an alert.
• Quarterly recalibration with a fresh 50-sample human-labeled batch. Models drift. Your traffic drifts. Your rubric drifts because the product changes. Recalibrating quarterly catches all three.

The first time I shipped this for the content pipeline, the judge caught a prompt change that improved a vanity metric (token efficiency) but tanked rubric-graded factual accuracy by 11 points. We would have shipped that change without the judge. That's the moment the system pays for itself.

A small table: what changed after calibration

This is from one of my own pipelines, same 150-sample gold set, before and after the calibration loop above.

The naive judge looks like a model. The calibrated judge looks like an evaluator. Same base LLM, same temperature. The entire delta is in the rubric, the few-shot examples, the position-flip protocol, and the gold-set-driven iteration.

What I'd do if I were starting today

If you're standing up an LLM judge for the first time on a production system, here is the order I'd do it in. No detours.

1. Build the 150-sample gold set first. Two days of work. Do not skip.
2. Start pairwise. Direct scoring later, only if you actually need an absolute number.
3. Cross-family judge. Never let a model grade its own homework.
4. Position-flip every comparison. Cheap insurance.
5. Read every disagreement on the first pass. Metrics lie about what's actually broken.
6. Wire it into CI before you start trusting the numbers. A judge that doesn't block bad changes is decoration.
7. Recalibrate quarterly. Set a calendar reminder. It will not happen otherwise.

Two things I'd avoid: rubrics with more than 5 criteria (judges get confused, humans get inconsistent), and any single-shot Likert scoring on a 10-point scale (you'll get 7s and 8s forever).

The deeper point: an LLM judge isn't a model, it's a measurement instrument. Like any instrument, it's worthless without calibration against a known reference. The gold set is the reference. Everything else is technique on top.

---

If you're building agentic workflows or RAG systems and your evals feel like they're lying to you, that's usually where I get the call. Happy to compare notes or help you set this up properly on your stack: lazar-milicevic.com/#contact. More from the same playbook on the blog.