DeepEval

Visit Website

Introduction

DeepEval is a Python library for testing LLM outputs. You write a test function, build an LLMTestCase with input and actual output, and assert a metric such as answer relevancy, faithfulness, or a custom rubric. The API is meant to feel like pytest: a failing metric fails the test. The project is Apache-2.0 at confident-ai/deepeval. A TypeScript package also exists.

Confident AI is the hosted product from the same group: datasets, tracing, and shared eval runs. It is optional. DeepEval runs without an account. Metric scores that use an LLM-as-judge consume that model’s API, which is a separate bill. DeepEval does not drive a browser.

Features

1. Pytest style: assert_test(test_case, [metric]) fails the pytest run when the metric score misses its threshold.
2. Built-in metrics: Answer relevancy, faithfulness, contextual precision and recall, hallucination, bias, and toxicity are published metrics. Each one documents what it needs (context, retrieval context, expected output).
3. G-Eval and custom metrics: A rubric you write is scored by a judge model when a built-in metric does not match the product.
4. Datasets: Cases can be loaded from JSON or from Confident AI datasets. A dataset run is a loop of test cases, not one prompt in a notebook.
5. Component tracing: @observe can trace nested LLM calls so an eval refers to a span rather than only the final string. Tracing uploads are a Confident AI feature when you want them stored.
6. CI: A normal pytest invocation is the gate. No separate runner is required.
7. Red teaming beside it: The DeepTeam package, also from Confident AI and Apache-2.0, covers adversarial tests. It is a different install from DeepEval’s metrics.
8. Providers: Judge and app models are configured with the usual vendor keys (OpenAI and others). The score depends on which judge you pinned.

Pros

1. Fits Python services: If the app is already tested with pytest, LLM checks are another test file rather than a new platform.
2. Thresholds are code: threshold=0.7 is reviewed in git. A dashboard default is not.
3. Retrieval metrics exist: Faithfulness and contextual precision apply to RAG pipelines, which is the case most teams actually ship.
4. Free library: Apache-2.0. You can run the full metric set without Confident AI.
5. The failure is a pytest failure: CI already knows how to surface it. No second reporting product is required.

Cons

1. Scores move: LLM-as-judge metrics are not deterministic. A threshold that passed yesterday can fail after a judge-model upgrade you did not pin.
2. Cost tracks tokens: Every metric call is a model call. A large dataset in CI needs a budget and a cache strategy.
3. Metric misuse: Faithfulness without a retrieval context, or relevancy without a real expected behaviour, produces a number that looks precise and is not.
4. Confident AI is a second product: Tracing, hosted datasets, and team workflow sit on confident-ai.com prices, not in the Apache license.
5. Python-first: The TypeScript package lags the Python docs. Java services are not the primary audience.

Usage and Links (Example)

pip install -U deepeval
export OPENAI_API_KEY="..."

import pytest
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_summary_is_relevant():
    test_case = LLMTestCase(
        input="What does DeepEval assert?",
        actual_output="DeepEval asserts metric scores inside pytest.",
    )
    assert_test(test_case, [AnswerRelevancyMetric(threshold=0.5)])

Run pytest. Pin the judge model in the metric or via DeepEval’s model config so CI does not drift to whatever “default” means next month. Do not point this at production customer data unless that is an approved use.

Resources:

• DeepEval website
• DeepEval docs
• GitHub: confident-ai/deepeval
• Confident AI pricing

Pricing

1. DeepEval: Free, Apache-2.0. No account is required to assert metrics locally or in CI. Judge-model tokens are paid to the model provider.
2. Confident AI Free: $0. The vendor lists 2 user seats, 1 project, 5 test runs per week, and 1 GB-month of trace spans. Extra runs lock; extra spans are dropped.
3. Confident AI Starter: $200 per month per organization. 5 projects, unlimited seats, 5 GB-months of trace spans, then $1 per GB-month. Adds online evals, annotation, and alerting.
4. Confident AI Team: $2,000 per month. Unlimited projects, 75 GB-months included, then $1 per GB-month, plus RBAC and SOC 2.
5. Enterprise: Custom, including on-prem and higher compliance options.

You can use DeepEval and never pay Confident AI. Pay Confident AI only if you want the hosted traces and shared runs.

Recommended for?

1. Python LLM apps whose authors already live in pytest.
2. RAG pipelines that can supply retrieved context and want a faithfulness check in CI.
3. Teams that want the eval in git, with thresholds reviewed like any other test.
4. Groups that outgrow local JSON results and are ready to compare Confident AI’s $200 Starter plan with running their own store.

Conclusion

DeepEval is the pytest-shaped way to put an LLM metric on a test case. Keep the library even if you skip Confident AI. Pin the judge model, treat the score as a signal with a threshold you chose, and do not confuse a green metric with a tested UI. Hosted collaboration starts at the Confident AI free limits and steps to $200 and $2,000 a month.