RAG Eval Metrics 2026: Ragas vs TruLens vs DeepEval

I've been running RAG systems in production for three years now, and the single most consistent lesson is this: teams overspend on retrieval and underspend on evaluation. They tune chunk sizes for a week, then ship with no way to detect when the pipeline regresses. When I audit a stuck RAG project, the first question I ask is not "which vector DB" but "how do you know when your answers are wrong?" Nine times out of ten, the answer is a Notion doc with 20 hand-picked questions.

This post is the reference I wish existed when I was picking an eval stack. Definitions with the actual formulas, then a head-to-head of the three frameworks I keep coming back to: Ragas, TruLens, and DeepEval, with a benchmark table from my own runs.

The four metrics that actually matter for RAG

Every serious RAG eval framework converges on the same core quartet. Learn these formulas and you can read any eval report intelligently.

Faithfulness measures whether the generated answer is grounded in the retrieved context. It's the primary defense against hallucination.

Faithfulness = (# claims in answer supported by context) / (# claims in answer)

The implementation is usually: an LLM extracts atomic claims from the answer, then a second LLM pass checks each claim against the retrieved chunks. Score of 1.0 means every claim is grounded. Below 0.85 in production and I start pulling logs.

Answer Relevancy measures whether the answer actually addresses the question, independent of correctness. The standard implementation generates N synthetic questions from the answer, then computes mean cosine similarity to the original question.

Answer Relevancy = mean(cosine_sim(original_q, generated_q_i)) for i in 1..N

Context Precision measures signal-to-noise in your retriever. Given the ground-truth answer, what fraction of retrieved chunks were actually useful, and were the useful ones ranked at the top?

Context Precision@K = sum(Precision@k * v_k for k in 1..K) / total_relevant

where v_k is 1 if chunk k is relevant, 0 otherwise. This is a ranked metric, so a system that puts the right chunk at position 8 out of 10 scores worse than one that puts it at position 1.

Context Recall measures whether the retriever pulled everything needed to answer the question. It decomposes the ground-truth answer into claims and checks how many are supported by the retrieved context.

Context Recall = (# ground-truth claims attributable to context) / (# ground-truth claims)

These four form a diagnostic grid. Low faithfulness with high context precision means your generator is inventing things. Low context recall with high faithfulness means your retriever is missing chunks and the generator is honestly answering with incomplete info. Low answer relevancy with high everything else usually means your prompt is off.

Ragas: fast to start, opinionated about metric design

Ragas is the framework I reach for first when a client wants a baseline eval running by end of week. It's opinionated, which is a feature. The metrics are defined, the formulas are published, and the LLM-as-judge prompts are versioned. You do not spend two days deciding what "faithfulness" means for your team.

The default flow:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

dataset = Dataset.from_dict({
    "question": questions,
    "answer": answers,
    "contexts": retrieved_contexts,
    "ground_truth": ground_truths,
})

result = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)

What I like: the synthetic testset generator. Ragas can build a diverse eval set from your documents with question types (simple, reasoning, multi-context, conditional) and difficulty distributions you control. For a client last quarter, I generated a 400-question testset from a 1,200-page product manual in about 40 minutes and $6 of API calls. That's dramatically faster than hand-writing.

What frustrates me: metric cost. Running the full quartet against 500 questions with GPT-4o as judge lands around $8-15 depending on context length. On CI runs against every PR, this adds up. My workaround is to run the full suite nightly and a cheaper subset (faithfulness + answer relevancy with a smaller judge) on PRs.

TruLens: strongest for observability and per-record debugging

TruLens takes a different posture. Where Ragas is about scoring a dataset, TruLens is about instrumenting a live app. It wraps your RAG chain, records every span, and computes feedback functions on each invocation. When something goes wrong in production, you can pull up the trace and see exactly which chunk was retrieved, what the prompt looked like, and which feedback function fired below threshold.

The killer feature is the "RAG Triad": Context Relevance, Groundedness (their name for faithfulness), and Answer Relevance, computed on every record. You can filter your dashboard to "answers with groundedness < 0.7 in the last 24 hours" and it becomes your debugging queue.

I've used TruLens on a customer-support RAG system where we needed to explain regressions to a non-technical stakeholder. The per-record trace view, with feedback scores highlighted red or green, was worth its weight. Ragas can compute the same metrics but does not give you that live drill-down out of the box.

Trade-off: TruLens has more surface area to learn. You need to wrap your app, define feedback providers, and manage the local database of records. For a five-day POC, this is overkill. For a system going into production with real users, it's the right investment.

DeepEval: pytest-native, best for CI and regression gates

DeepEval is the newest of the three and the one I recommend when the team is engineering-heavy and wants evals to feel like unit tests. It's built on pytest. You write test cases, assert metric thresholds, and run deepeval test run. Failed evals become failed CI builds.

from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric, ContextualPrecisionMetric
from deepeval.test_case import LLMTestCase

def test_rag_faithfulness():
    test_case = LLMTestCase(
        input="What is the refund window?",
        actual_output=rag_answer,
        retrieval_context=retrieved_chunks,
        expected_output="30 days from purchase",
    )
    assert_test(test_case, [
        FaithfulnessMetric(threshold=0.9),
        ContextualPrecisionMetric(threshold=0.8),
    ])

DeepEval also ships a broader metric catalog than the other two: G-Eval (custom criteria via LLM-as-judge), bias, toxicity, hallucination, task completion, and a growing set of agent-specific metrics like tool correctness. If you're building agentic RAG, this matters.

The weakness is dataset-level reporting. Ragas is more mature for "score this whole eval set and give me aggregates." DeepEval's Confident AI cloud dashboard closes that gap but pushes you toward a paid tier.

Benchmark: same RAG system, three frameworks

I ran an apples-to-apples benchmark on a RAG system I built for internal use: a 4,800-document technical knowledge base, hybrid search (BM25 + pgvector with reciprocal rank fusion), Claude Sonnet 4 as the generator, 150 evaluation questions with hand-labeled ground truth. Judge model: GPT-4o for all three frameworks to eliminate judge variance.

Three observations from this run:

The absolute scores agree within 3-4%. Faithfulness scores of 0.876, 0.891, and 0.902 for the same system tell me the underlying formulas are converging. Do not obsess over which framework "gave a higher score." The signal is directional: all three flagged the same 12 questions as low-faithfulness outliers, and 9 of those 12 turned out to be genuine retrieval failures on ambiguous queries.

Ragas is fastest for batch scoring. If your workflow is "run a suite, get a number," Ragas wins on wall clock and setup time.

TruLens costs more because it captures more. The extra latency and cost come from recording full traces. That's exactly what you want in production and exactly what you do not want for a quick CI check.

The stack I actually run

I stopped picking one framework and started using two. On every production RAG project I own, the setup looks like this:

1. DeepEval in CI. Twenty to fifty golden test cases with hard thresholds. If faithfulness on the golden set drops below 0.85 or context precision drops below 0.75, the build fails. This catches regressions from prompt changes, model swaps, and chunking tweaks.

2. Ragas nightly on a 300-500 question synthetic testset. Regenerated monthly from the current document corpus. Aggregates go to a dashboard. This catches slow drift as the knowledge base grows and query distribution shifts.

3. TruLens on the live app for the first 90 days after launch, then either kept or downgraded to sampled tracing depending on volume. This is where I actually debug user complaints.

Total added infra cost for a mid-size RAG deployment: around $80-150/month in judge API calls, plus a small Postgres instance for TruLens records. For a system whose failure mode is confidently wrong answers to paying customers, this is not a real cost.

What I'd do if I were starting today

If you're building a RAG POC this week, install Ragas, generate a synthetic testset from your docs, and run the four core metrics. You'll have a baseline in an afternoon.

If you're taking a RAG system into production next quarter, add DeepEval test cases to CI on day one. Retrofitting evals into a system with real users is painful. Building the test harness alongside the code is easy.

If you already have a production RAG system and no evals, start with TruLens. Wrap the app, let it record for a week, then look at your groundedness distribution. You will find surprises. Everyone does.

The mistake I see most: teams pick a framework based on GitHub stars and never look at the actual metric formulas. All three frameworks I compared are solid engineering. What matters is that you understand what faithfulness of 0.87 means for your business, what threshold triggers a rollback, and who gets paged when the nightly run fails.

If you're stuck on the eval side of a RAG rollout, or you want a second opinion on whether your metrics are catching what they should, get in touch at lazar-milicevic.com/#contact. More on production RAG architecture and agent evals over on the blog.