RAG Eval Metrics 2026: Ragas vs TruLens vs DeepEval

I've been running RAG systems in production for a long time, and the single most consistent lesson is this: teams overspend on retrieval and underspend on evaluation. They tune chunk sizes for a week, then ship with no way to detect when the pipeline regresses. When I audit a stuck RAG project, the first question I ask is not "which vector DB" but "how do you know when your answers are wrong?" Nine times out of ten, the answer is a Notion doc with 20 hand-picked questions.

This post is the reference I wish existed when I was picking an eval stack. Definitions with the actual formulas, then a head-to-head of the three frameworks I keep coming back to: Ragas, TruLens, and DeepEval, with illustrative results from my own runs and a cross-check against a larger independent benchmark published in 2026. If you're searching for "how to evaluate a RAG pipeline" or "Ragas vs TruLens vs DeepEval," start with the quick comparison below, then read the sections that match your situation.

Quick answer: which RAG evaluation framework should you pick?

• Ragas if you need a baseline number fast and want synthetic test data generation for RAG out of the box.
• TruLens if you need production observability, per-record traces and an audit trail.
• DeepEval if you want evals to behave like pytest unit tests and gate your CI.
• More than one if the system serves paying customers. I run all three in different roles (details in "The stack I actually run").

One caveat before the details: these tools do not share one metric set, and the same metric name can mean different math in different libraries. Ragas, TruLens and DeepEval overlap, but they are not interchangeable.

The metrics that actually matter for RAG

The most common core set in RAG evaluation is Ragas's quartet: faithfulness, answer relevancy, context precision and context recall. TruLens and DeepEval organize things differently, and I'll map them below. Learn these formulas and you can read any eval report intelligently.

Faithfulness measures whether the generated answer is grounded in the retrieved context. It's the primary defense against hallucination, and the core of any RAG hallucination detection setup.

Faithfulness = (# claims in answer supported by context) / (# claims in answer)

In Ragas this ranges from 0 to 1 and needs no reference answer. The implementation is usually: an LLM extracts atomic claims from the answer, then a second LLM pass checks each claim against the retrieved chunks. A score of 1.0 means every claim is grounded. Below 0.85 in production and I start pulling logs. That is my own rule of thumb, not official guidance.

DeepEval's FaithfulnessMetric documents a similar two-step design: extract claims from actual_output, then classify each claim as truthful or not against retrieval_context. Its formula is the number of truthful claims divided by the total number of claims. A low score is treated as a signal that the prompt or the model is inventing unsupported claims, not that the retriever is broken. That distinction matters for how you route the fix.

Two subtleties that bite people when comparing tools:

• Support vs. contradiction. DeepEval's faithfulness emphasizes detecting contradictions between the output and the retrieved context, while Ragas's is support-based. Scores from the two are not directly comparable.
• Borderline claims. In the DeepEval release I checked, claims the context neither confirms nor denies are labeled borderline and count as faithful unless penalize_ambiguous_claims is set, which is off by default. If your faithfulness numbers look suspiciously good, check this flag, and verify the behavior against the DeepEval documentation for the version you pin.

Answer Relevancy measures whether the answer actually addresses the question, independent of correctness. The Ragas implementation generates artificial questions from the response (3 by default), then averages the cosine similarity between each one and the user input.

Answer Relevancy = mean(cosine_sim(original_q, generated_q_i)) for i in 1..N

The Ragas docs warn that the score is not guaranteed to stay within 0 to 1, because cosine similarity ranges from -1 to 1. Don't be alarmed by an odd value, but do clip or inspect it in aggregation. DeepEval takes a different route: it uses LLM verdicts on each statement and no embeddings, so the same metric name hides different math.

Context Precision measures signal-to-noise in your retriever. Given the ground-truth answer, what fraction of retrieved chunks were actually useful, and were the useful ones ranked at the top?

Context Precision@K = sum(Precision@k * v_k for k in 1..K) / total_relevant

where v_k is 1 if chunk k is relevant, 0 otherwise. This is a ranking-aware metric, so a system that puts the right chunk at position 8 out of 10 scores worse than one that puts it at position 1. The older Ragas v0.1.x docs describe it as Average Precision. Ragas' LLM-based implementation compares each chunk in retrieved_contexts against the reference answer to judge relevance. Note that I have not confirmed that the exact formula on the current docs page matches the one above, and other tools define precision differently (for example, relevant chunks divided by k), so check the docs for the version you pin before comparing numbers across tools.

Context Recall measures whether the retriever pulled everything needed to answer the question. It decomposes the ground-truth answer into claims and checks how many are supported by the retrieved context. Ragas documents this explicitly: the LLM-based version is the number of reference claims supported by the retrieved context divided by the total number of reference claims, and it always needs a reference answer.

Context Recall = (# ground-truth claims attributable to context) / (# ground-truth claims)

These four form a diagnostic grid. Low faithfulness with high context precision means your generator is inventing things. Low context recall with high faithfulness means your retriever is missing chunks and the generator is honestly answering with incomplete info. Low answer relevancy with high everything else usually means your prompt is off.

How the three frameworks name things

This is the part most comparison posts skip. The metric families overlap but are not identical:

TruLens does not use the four Ragas metrics. Its signature evaluation is the RAG Triad of context relevance, groundedness and answer relevance. DeepEval's RAG set has five metrics: the first two (answer relevancy, faithfulness) evaluate the generator, and the last three (contextual precision, recall, relevancy) evaluate the retriever.

How to read a metric report without fooling yourself

Four things I now check before I trust any headline number:

1. Is the metric reference-based or reference-free? Faithfulness and answer relevancy do not need a ground-truth answer. Context recall does. If your "eval" is running only reference-free metrics because you never wrote ground truths, you are measuring internal consistency, not correctness.
2. What is the judge model, and is it the same as the generator? Using the same model to generate and to judge inflates scores. I fix the judge model per project and freeze it. This is the heart of any LLM-as-judge setup: the judge is a measurement instrument, and changing it changes the ruler. When I have to change judges, I re-baseline everything so trend lines stay comparable.
3. What happens on edge cases? If Ragas statement extraction yields zero statements, the faithfulness score is undefined, often NaN. A naive mean() over a column with NaNs will either crash or silently drop your hardest rows. Decide explicitly how to aggregate these.
4. What does the pass threshold actually mean? In DeepEval every metric outputs a score from 0 to 1, and the default pass threshold is 0.5 for every metric. That is a very lenient default for faithfulness. Set thresholds deliberately per metric and per use case rather than inheriting the default.

Ragas: fast to start, opinionated about metric design

Ragas is the framework I reach for first when a client wants a baseline eval running by end of week. It's opinionated, which is a feature. The metrics are defined, the formulas are published, and the LLM-as-judge prompts are versioned. You do not spend two days deciding what "faithfulness" means for your team. It's open source, so there's little commercial rug-pull risk on the core (check the project's GitHub repository for the license and current maintainers). The approach is described in the original paper, Ragas: Automated Evaluation of Retrieval Augmented Generation, first submitted in September 2023, which presents it as a framework for reference-free evaluation of RAG pipelines.

The classic flow, which you'll still see in most tutorials:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

dataset = Dataset.from_dict({
    "question": questions,
    "answer": answers,
    "contexts": retrieved_contexts,
    "ground_truth": ground_truths,
})

result = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)

API migration warning. Ragas v0.4 moves to an experiment-based architecture, and the v0.3 to v0.4 migration guide lists the old evaluate() function as deprecated. That means a lot of older tutorial code, including the snippet above, may need updating. Check PyPI for the current release before you pin. If you're standing up Ragas today, pin your version explicitly and plan a migration window. I've seen teams pin ragas==0.1.x in a pyproject.toml, forget about it, and get bitten a year later when a new dev runs pip install -U.

What I like: the synthetic testset generator. Ragas includes a generator for synthetic test sets that builds an eval set from your documents, so you don't have to hand-write every question. The exact options and question types depend on the version you install, so check the docs for your pinned release. On a recent client engagement I generated a several-hundred-question testset from a large product manual in under an hour of wall time, dramatically faster than hand-writing. It's the fastest path from "no evals" to "a defensible baseline" I know of. One rule I never skip: have a human spot-check a sample of the synthetic questions and ground truths before trusting them. Synthetic test data generation for RAG is a bootstrap, not a substitute for real user queries; once you have production traffic, mine real questions into the set.

What frustrates me: metric cost. In my own experience, running the full quartet against a few hundred questions with a frontier-class judge is not free, and on CI runs against every PR the judge-call bill adds up quickly. My workaround is to run the full suite nightly and a cheaper subset (faithfulness + answer relevancy with a smaller judge) on PRs. Your mileage will vary with context length, judge choice and retry policy; measure a small pilot batch before committing to a cadence.

TruLens: strongest for observability and per-record debugging

TruLens takes a different posture. Where Ragas is about scoring a dataset, TruLens is about instrumenting a live app. It wraps your RAG chain, records every span, and computes feedback functions on each invocation. When something goes wrong in production, you can pull up the trace and see exactly which chunk was retrieved, what the prompt looked like, and which feedback function fired below threshold.

The killer feature is the RAG Triad: Context Relevance, Groundedness (their name for faithfulness), and Answer Relevance. The logic is neat: if all three legs score well, the app is very likely free of hallucination; if one fails, the failing leg tells you where to look. Reviewing recorded traces alongside their feedback scores gives you a natural debugging queue: start with the records where groundedness came out low and work from there.

Governance note. TruLens is an open-source project that has been associated with TruEra and, according to third-party sources, is now maintained under Snowflake. That ownership claim comes largely from third-party sources, so confirm project governance, licensing and self-hosting terms on trulens.org and in the GitHub repository before you rely on it in a procurement conversation. If you want the deeper Snowflake integration (tracing and application registration for Snowflake AI Observability), check Snowflake's documentation for the exact package names and minimum versions. If you're not on Snowflake, none of that should be forced on you.

I've used TruLens on a customer-support RAG system where we needed to explain regressions to a non-technical stakeholder. The per-record trace view, with feedback scores highlighted red or green, was worth its weight. Ragas can compute the same kind of metrics but does not give you that live drill-down out of the box. If you need to justify the choice to a skeptical stakeholder, look at the project's official site and GitHub repository for its current adopter list and community activity rather than relying on a snapshot in a blog post (star counts change constantly).

Trade-off: TruLens has more surface area to learn. You need to wrap your app, define feedback providers, and manage the local database of records. For a five-day POC, this is overkill. For a system going into production with real users, it's the right investment.

DeepEval: pytest-native, best for CI and regression gates

DeepEval is the one I recommend when the team is engineering-heavy and wants evals to feel like unit tests. It's built on pytest. You write test cases, assert metric thresholds, and run deepeval test run. Failed evals become failed CI builds.

from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric, ContextualPrecisionMetric
from deepeval.test_case import LLMTestCase

def test_rag_faithfulness():
    test_case = LLMTestCase(
        input="What is the refund window?",
        actual_output=rag_answer,
        retrieval_context=retrieved_chunks,
        expected_output="30 days from purchase",
    )
    assert_test(test_case, [
        FaithfulnessMetric(threshold=0.9),
        ContextualPrecisionMetric(threshold=0.8),
    ])

Note the explicit thresholds. As covered above, DeepEval's default pass threshold is 0.5, which is too lenient to catch most regressions, so always set your own.

DeepEval's open-source RAG suite covers Faithfulness, Contextual Recall, Contextual Precision, and Contextual Relevancy as separate metrics, plus Answer Relevancy. It also ships a composite RAGAS metric defined as the average of answer relevancy, faithfulness, contextual precision, and contextual recall, which is useful when a product manager wants a single dashboard number and dangerous if you use it to hide a low faithfulness behind a high recall. Whenever you average metrics, keep the per-metric scores visible next to the composite.

Beyond RAG, DeepEval documents broader coverage than Ragas: RAG metrics (faithfulness, contextual recall/precision/relevancy), agentic metrics (task completion, tool correctness), and conversational metrics. See the project's GitHub README and metrics documentation for the current list. If you're building agentic RAG (retrieval plus tool calls plus multi-step reasoning), this matters. A retrieval-only eval will silently miss half the failure modes.

The weakness is dataset-level reporting. Ragas is more mature for "score this whole eval set and give me aggregates." DeepEval also offers a hosted cloud dashboard (Confident AI) that can help close that gap; I haven't verified its current plans or pricing, so check the vendor's site before deciding whether it fits your budget.

Benchmark: same RAG system, three frameworks (illustrative, my own internal run)

The numbers below come from an internal benchmark I ran on a RAG system I built for my own use. They are not a public dataset and they are not intended as a general claim about the frameworks. Treat them as one engineer's ballpark from one run. The setup: a mid-sized technical knowledge base, hybrid search (BM25 + pgvector with reciprocal rank fusion), an LLM generator, and a hand-labeled ground-truth set of a few hundred evaluation questions. Judge model: the same GPT-4-class model across all three frameworks to eliminate judge variance. The configuration is illustrative; I'm not presenting it as a reproducible published benchmark.

Three observations from this run:

The absolute scores landed close together. For the same system, the frameworks converged on similar faithfulness numbers. Given the definitional differences above (support-based vs. contradiction-based), I wouldn't expect that to hold on every dataset, so don't obsess over which framework "gave a higher score." The signal is directional: all three flagged the same handful of questions as low-faithfulness outliers, and most of those turned out to be genuine retrieval failures on ambiguous queries.

Ragas was fastest for batch scoring. If your workflow is "run a suite, get a number," Ragas wins on wall clock and setup time in my hands.

TruLens costs more because it captures more. The extra latency and cost come from recording full traces. That's exactly what you want in production and exactly what you do not want for a quick CI check.

Cross-check against a larger independent benchmark

A run on my own system is enough to differentiate frameworks operationally, but it's too small to make general claims about which one "wins." For that, I cross-checked against a much larger public run.

An AIMultiple benchmark tested five RAG evaluation tools (Weights & Biases Weave, Ragas, TruLens, DeepEval and UpTrain) across 1,460 questions and more than 14,600 scored contexts under identical conditions using GPT-4o as judge. Findings worth internalizing before you pick a stack:

• TruLens ranked first of the five tools on three of four ranking metrics (NDCG@5, Spearman ρ, and MRR), ahead of WandB Weave, Ragas, DeepEval and UpTrain. If your job is "distinguish good retrieval from bad retrieval," TruLens's scorer discriminates well. Read the original benchmark for the exact methodology and numbers.
• Under standard (non-adversarial) conditions, WandB, TruLens and Ragas were statistically tied on Top-1 Accuracy, with 95% confidence intervals overlapping between 94.0% and 98.0%. Translation: at the top of the ranking, the three leaders are indistinguishable within noise. Do not switch frameworks over a 1% delta.
• DeepEval's statement-decomposition approach produced a competitive NDCG@5 of 0.923 but scored golden (correct) contexts at a mean of only 0.46, versus 0.82–0.91 for the other tools. It's weaker at identifying the single best context, which matches what I've seen: DeepEval is stronger as a pass/fail regression gate than as a fine-grained retrieval ranker.

And the finding that should worry anyone shipping RAG to customers: comparisons of LLM evaluation frameworks report that these tools struggle to distinguish a factually wrong context from a factually correct one. The AIMultiple data is more nuanced, with TruLens showing the best discrimination of the group, but even the best performer was weak at this task. Treat all of these tools as having a serious blind spot here. If the retriever fetches a plausible-looking chunk with wrong facts, the eval frameworks will happily score the resulting answer as "faithful," because the answer is faithful to the retrieved chunk, which is the actual formula. Faithfulness is not truthfulness. This is why I always pair automated metrics with a small hand-labeled truthfulness set that a human reviews weekly.

A worked example: diagnosing a bad answer with the metrics

Here is how I use the grid on a real-shaped failure. The numbers below are made up with round values purely to show the reasoning, not results from a real system.

Say a support bot is asked: "What is the refund window for annual plans?" The ground truth is "30 days from purchase." The bot answers: "Annual plans can be refunded within 60 days."

Now imagine the three plausible diagnostic outcomes:

• Case A: faithfulness 0.2, context precision 0.9, context recall 1.0. The retriever did its job; the right policy chunk was retrieved and ranked high. The generator ignored it and invented "60 days." Fix the prompt (stronger grounding instructions), try a different model, or lower temperature. Don't touch the retriever.
• Case B: faithfulness 1.0, context precision 0.3, context recall 0.0. The answer is perfectly faithful to what was retrieved, but the retrieved chunk was an old policy page that said 60 days. The generator is honest and the retriever is wrong. Fix chunking, metadata filters (document version), or reranking. This is also the failure mode that the benchmarked tools struggled to catch when the wrong context looks plausible, so a human-labeled truthfulness set is your safety net here.
• Case C: answer relevancy 0.3, faithfulness 0.9. The bot confidently answered a different question (say, the monthly plan refund policy). Look at query understanding and the prompt template.

The point is that a single blended "RAG score" would have told you nothing in any of these cases. Per-metric scores route the fix to the right component, which is the entire reason to evaluate retrieval and generation separately.

The stack I actually run

I stopped picking one framework and started using two or three. On every production RAG project I own, the setup looks like this:

1. DeepEval in CI. Twenty to fifty golden test cases with hard thresholds. If faithfulness on the golden set drops below 0.85 or context precision drops below 0.75, the build fails. Those thresholds are my own opinion, tuned per project, not vendor guidance. This catches regressions from prompt changes, model swaps, and chunking tweaks.

2. Ragas nightly on a 300-500 question synthetic testset. Regenerated monthly from the current document corpus. Aggregates go to a dashboard. This catches slow drift as the knowledge base grows and query distribution shifts.

3. TruLens on the live app for the first 90 days after launch, then either kept or downgraded to sampled tracing depending on volume. This is where I actually debug user complaints.

4. A small, curated truthfulness set reviewed by a human weekly. Ten to thirty questions where the ground-truth answer is known and important enough that being wrong is a business incident. This is the layer that catches what all three frameworks miss: the "retrieved chunk is wrong but answer looks faithful" failure mode.

In my experience the added infra cost for this kind of setup on a mid-size RAG deployment is modest compared to the value: a monthly line of judge API calls plus a small Postgres instance for TruLens records. Actual figures depend heavily on your traffic, judge model and retention policy. For a real number, run the pilot for a week and read your own bill (AWS console, Anthropic/OpenAI usage dashboards) before locking anything in. For a system whose failure mode is confidently wrong answers to paying customers, this is not a real cost.

Practical rules for running RAG in production

A few habits that have saved me, beyond the framework choice:

• Freeze and version the judge. Record the judge model, its version and the metric library version alongside every score. Without that, a trend line is meaningless.
• Separate retriever and generator dashboards. Retriever metrics (context precision, recall, relevancy) and generator metrics (faithfulness, answer relevancy) should never be blended into one headline number.
• Sample production traffic into your eval set. Synthetic questions get you started; real user queries keep you honest. Review a slice of low-scoring production traces every week and promote the interesting ones into your golden set.
• Handle undefined scores explicitly. Count NaNs as their own failure category instead of letting aggregation drop them.
• Keep thresholds in version control. Treat threshold changes like code changes, with a reviewer and a reason.

Decision matrix by situation

If it helps, here is how I actually choose in an intake call:

• Solo engineer, one-week POC, need a baseline number. Ragas only. Skip the rest until you have real users.
• Small team, going to production in 4–8 weeks, engineering-led. DeepEval in CI from day one, Ragas nightly, TruLens added at launch.
• Enterprise team, regulated industry, stakeholders need audit trails. TruLens first (traces are the audit trail), DeepEval in CI, Ragas for periodic aggregate reports. If you're already on Snowflake, check Snowflake's documentation for the AI Observability integration, which can be a real accelerant.
• Agentic RAG with tool calls. DeepEval, because of the agentic metrics (task completion, tool correctness). Retrieval-only frameworks will miss half your failure modes.

What I'd do if I were starting today

If you're building a RAG POC this week, install Ragas (pinned to a specific version, and check which API style your version expects), generate a synthetic testset from your docs, and run the core metrics. You'll have a baseline in an afternoon.

If you're taking a RAG system into production next quarter, add DeepEval test cases to CI on day one. Retrofitting evals into a system with real users is painful. Building the test harness alongside the code is easy.

If you already have a production RAG system and no evals, start with TruLens. Wrap the app, let it record for a week, then look at your groundedness distribution. You will find surprises. Everyone does.

The mistake I see most: teams pick a framework based on GitHub stars and never look at the actual metric formulas. All three frameworks I compared are solid engineering. What matters is that you understand what a faithfulness of 0.87 means for your business, what threshold triggers a rollback, and who gets paged when the nightly run fails. And, because the independent comparisons above show that these tools are weak at catching "wrong-but-plausible context," that you have at least one human-reviewed truthfulness gate that no LLM judge is allowed to bypass.

FAQ

What is the best RAG evaluation framework in 2026? There isn't a single winner. Ragas is the fastest path to a baseline and has strong synthetic test data generation, TruLens is strongest for production tracing, and DeepEval is the most natural fit for CI regression gates. In the independent benchmark above, the top tools were statistically tied on top-1 accuracy, so pick based on workflow, not on a 1% score difference.

Is faithfulness the same as truthfulness? No. Faithfulness checks whether the answer is supported by the retrieved context, not whether the context itself is correct. A wrong chunk can produce a perfectly faithful, perfectly wrong answer. Pair automated metrics with a human-reviewed truthfulness set.

Can I compare scores between Ragas and DeepEval? Not directly. The same metric name can have different math: Ragas answer relevancy uses embeddings and generated questions, DeepEval uses LLM verdicts on statements, and faithfulness is support-based in one and contradiction-focused in the other. Compare trends within a tool, not absolute numbers across tools.

How do I evaluate a RAG pipeline without ground-truth answers? Start with reference-free metrics (faithfulness, answer relevancy, context relevance), then build ground truths gradually from reviewed production traffic and spot-checked synthetic data. Be aware that reference-free metrics measure internal consistency, not correctness.

If you're stuck on the eval side of a RAG rollout, or you want a second opinion on whether your metrics are catching what they should, get in touch at lazar-milicevic.com/#contact. More on production RAG architecture and agent evals over on the blog.