RAG Evaluation Metrics: A 12-Point Checklist (2026)

A support RAG pipeline I shipped last year scored 0.91 on answer relevancy and 0.88 on faithfulness in my offline eval suite. Two weeks into production it confidently returned the wrong account number to a customer query — pulled from a stale doc, formatted plausibly, cited a real source chunk. The eval said it was fine. The user saw a hallucinated 9-digit number.

That incident forced me to rebuild how I evaluate retrieval-augmented systems. The standard trio — context precision, answer relevancy, faithfulness — is necessary but nowhere near sufficient for production. Below is the 12-metric checklist I now run on every RAG system I deliver, what each one catches that the others miss, and the thresholds I actually use.

Why the standard 3-metric eval misses production failures

The default RAGAS-style setup — answer relevancy, faithfulness, context precision — measures whether the model used the retrieved context coherently. It does not measure whether the retrieved context was the right context, whether the answer is factually correct against ground truth, or whether the system should have answered at all. My 0.91 case scored well because the model faithfully grounded its answer in a chunk that happened to contain the wrong number. Faithfulness ≠ correctness. That distinction is the entire reason the checklist below exists.

A production RAG system fails along four axes: retrieval, generation, behavior under uncertainty, and drift over time. You need at least one metric per axis, ideally three.

The 12-point checklist

I group the metrics by the failure mode they actually catch, not by where they sit in the pipeline. Here is the full list with the thresholds I gate releases on for a mid-stakes B2B support/knowledge use case. YMYL or financial systems need stricter numbers.

The rest of this post goes deep on the four that most teams skip: answer correctness, refusal correctness, entity-level hallucination, and drift.

Retrieval metrics: recall is the one that matters

Context Recall @ k

For every gold question, you need a labeled ground-truth chunk (or set of chunks) that contains the answer. Context recall @ k asks: was the right chunk in the top-k retrieved? If recall @ 10 is below 0.90, no amount of prompt engineering downstream will save you. The model cannot reason about what it never sees.

How I build the ground-truth set: 150–300 real user queries from logs (or synthetic ones generated from the corpus with Claude, then human-reviewed), each tied to the specific document and chunk that answers it. This is the single highest-leverage artifact in a RAG project and the one most teams skip.

Context Precision @ k and MRR

Precision @ k tells you how much noise the LLM is wading through. MRR tells you whether the right answer is in position 1 or position 9. Both matter because context window is finite and attention to mid-context chunks is measurably worse than to the first or last.

If recall is high but MRR is low, the fix is reranking — usually a cross-encoder (BGE, Cohere Rerank) on the top 30 → top 5. In a hybrid search setup (BM25 + dense + RRF), I see MRR jump from ~0.62 to ~0.84 with a reranker added. That single change is usually worth more than swapping embedding models.

Generation metrics: faithfulness is not correctness

Answer Correctness vs gold

This is the metric that would have caught my account-number incident. You score the generated answer against a human-written correct answer for each gold question, not against the retrieved context. I use an LLM-as-judge here (Claude Sonnet, with a calibrated rubric — see my earlier post on building LLM judges), with a small human-labeled validation set to keep the judge honest.

A faithful answer can be wrong if the retrieved chunk is wrong, outdated, or misleadingly close to the question. Faithfulness measures internal consistency; correctness measures truth. Ship both.

Citation Accuracy

If your system surfaces citations to users (and it should), measure whether the citation actually supports the specific claim. The failure mode: the model writes a three-sentence answer, cites chunk #4, but only sentence two is supported by chunk #4 — sentences one and three are paraphrased from training data. Users trust citations. Wrong citations are worse than no citations.

I implement this as a per-sentence check: for each sentence with a citation, ask the judge "does this exact chunk entail this exact sentence?" Score the ratio.

Entity-level Hallucination Rate

This is the metric I added specifically because of the account-number incident. Aggregate scores hide entity-level errors. A response can be 95% faithful and still contain one fabricated number that costs you the customer.

What I do: extract all named entities, numbers, dates, identifiers, and proper nouns from the generated answer. For each, check whether it appears verbatim (or in a normalized form) in the retrieved context. Anything in the answer that is not in the context is a candidate hallucination. Flag it, log it, and in high-stakes flows, block the response.

# simplified entity-grounding check
def entity_hallucination_score(answer: str, context: str) -> float:
    entities = extract_entities(answer)  # NER + regex for numbers/IDs
    grounded = sum(1 for e in entities if normalized_in(e, context))
    return 1.0 - (grounded / max(len(entities), 1))

For a financial or healthcare RAG, I run this as a hard gate at inference time, not just at eval time. If hallucination_score > 0, the response goes to a fallback ("I cannot confirm this — please verify with [source]") instead of to the user.

Behavioral metrics: when should the system refuse?

Refusal Correctness

A RAG system that answers everything is broken. A good one knows when its retrieved context is insufficient and says so. I build a deliberate "unanswerable" set into my eval: 20% of gold questions have no supporting chunk in the corpus, by design. The metric is:

• True refusal rate: % of unanswerable questions where the system abstained
• False refusal rate: % of answerable questions where the system wrongly abstained

The trade-off curve here is the single most important business decision in a RAG deployment. A customer-facing legal assistant should sit at 95%+ true refusal even at the cost of 15% false refusals. A casual internal search tool can be the opposite. Make this decision explicit, with the stakeholder, in writing. Then tune the confidence threshold (often a retrieval-score floor combined with a self-check prompt) to hit the chosen point.

Operational metrics: the ones that get ignored until they break you

Latency P95

Mean latency lies. P95 is what your users actually feel. Hybrid search + reranking + LLM generation + citation post-processing can quietly drift from 2.1s to 4.8s as your corpus grows. I set a P95 budget at design time and alert on it. Common P95 killers: cross-encoder reranking on too many candidates, sequential rather than parallel embedding + BM25, synchronous reembedding of stale chunks during query time.

Cost per Resolved Query

A query is "resolved" if it produced a correct, non-refused answer the user did not retry. Cost includes embedding, retrieval, reranking, and generation tokens. For one ContentStudio pipeline I tracked, switching from GPT-4-class generation to Claude Haiku for the first-pass with selective escalation cut cost-per-resolved-query by 71% with a 1.2% drop in correctness — net positive for that use case. You cannot make that decision without the metric.

Drift Score

Corpora change. Embeddings of new documents may cluster differently. User questions shift. I run the full gold-set eval weekly and alert on any metric dropping >5% week-over-week. I also track distributional drift on retrieval scores — if the average top-1 similarity score for production traffic drops 10%, something has changed in either the index or the query distribution, and I want to know before the correctness metric catches it.

The eval harness itself: how I actually run this

A few specifics from how I have this wired in production:

1. Gold set is versioned, in the repo, with the corpus snapshot it was labeled against. When the corpus changes materially, the gold set gets reviewed.
2. Two judges, not one. I use Claude as the primary judge and a smaller, cheaper model as a tie-breaker on borderline cases. Disagreements >10% get sampled for human review.
3. CI gate on every PR that touches retrieval, prompts, or models. Below-threshold runs block merge.
4. Production shadow eval. A sampled 1% of real queries gets re-run through the eval pipeline with an LLM judge against the live response. This catches the things gold sets cannot — novel queries, fresh edge cases.
5. Per-segment breakdown. Aggregate scores hide segment failures. I always report metrics sliced by document type, query length, and topic cluster. A 0.87 average can hide a 0.42 on the segment where 30% of your revenue lives.

What I'd do if I were starting today

Build the gold set first. Before you pick an embedding model, before you write a prompt, before you choose a vector DB. 200 real questions with labeled answer chunks beats every benchmark on every leaderboard. Without it you are guessing.

Then wire up metrics 1, 4, 5, 8, and 9 from the table above. Those five catch ~80% of production failures. Add the others as your system matures and your stakes grow. Do not chase 0.99 on faithfulness while ignoring entity-level hallucination — that is exactly the trap that bit me.

Finally: never trust a single aggregate score. The number that matters is the worst-segment score on the metric most aligned with business impact. Everything else is dashboard candy.

---

If you are building or operating a RAG system and want a second pair of eyes on the eval harness — or if your offline scores look great and production keeps surprising you — I help teams ship this kind of thing for a living. Reach out at lazar-milicevic.com/#contact, or read more on the blog where I write about how I build the systems behind these metrics.