Promptfoo vs DeepEval vs Langfuse: Which LLM Eval Tool?

Last updated 30 October 2026, including Promptfoo joining OpenAI and Langfuse joining ClickHouse.

Promptfoo, DeepEval and Langfuse are three of the most popular open-source tools for testing LLM applications. They’re often compared as if they were rivals, but they solve different parts of the problem. Promptfoo is a command-line eval and red-teaming tool. DeepEval is a Python testing library with a large set of metrics. Langfuse is an observability platform that records traces and runs evaluations on them.

This guide compares them on purpose, licence, language, metrics, red teaming, tracing, CI fit, cost and ownership, and ends with practical recommendations. Many teams end up using two of the three together.

Promptfoo vs DeepEval vs Langfuse at a glance

	Promptfoo	DeepEval	Langfuse
What it is	CLI for evals and red teaming	Python library for LLM tests and metrics	Platform for traces, prompts and evals
Main language	TypeScript (CLI, YAML config)	Python (also a TypeScript package)	TypeScript server; Python and JS SDKs
Licence	MIT	Apache-2.0	MIT for core features
GitHub stars (30 Oct 2026)	~26,600	~19,200	~36,800
Latest release (30 Oct 2026)	0.125.1	python-v4.2.4	v4.72.0
Best at	Comparing prompts and models; red teaming	Pytest-style tests with research-backed metrics	Production tracing and evaluating real traffic
Hosted option	Enterprise plan	Confident AI	Langfuse Cloud
Owner (Oct 2026)	OpenAI	Confident AI	ClickHouse

Star counts and releases come from the GitHub API on 30 October 2026. GitHub reports Langfuse’s licence as “NOASSERTION” because the repository mixes MIT-licensed core code with separately licensed enterprise code. ClickHouse says Langfuse’s core features remain under the existing MIT licence.

Promptfoo: evals and red teaming from the command line

Promptfoo evaluates prompts and LLM applications from a YAML config. You declare providers (models or your own app), prompts and tests, and promptfoo eval runs the matrix and shows a pass/fail table.

What Promptfoo does well

• Comparing prompts and models side by side. Add a second provider and you get a comparison, not a second project.
• Assertions in config. Tests can use equals, contains, regex, similarity or model-graded rubrics.
• Red teaming. promptfoo redteam generates adversarial cases such as prompt injections and jailbreaks against your target.
• CI gating. The eval exits non-zero when assertions fail, so a pull request can be blocked by a prompt change.
• Local first. Evals run on your machine, and promptfoo view serves results locally.
• App targets. You can point it at your own HTTP endpoint, so tests cover retrieval and tools as well as the model.

A minimal config looks like this:

# promptfooconfig.yaml
prompts:
  - "Summarise this support ticket in one sentence: {{ticket}}"
providers:
  - openai:gpt-4.1-mini
  - anthropic:messages:claude-sonnet-4-5
tests:
  - vars:
      ticket: "My invoice for March was charged twice."
    assert:
      - type: icontains
        value: "twice"
      - type: llm-rubric
        value: "Mentions a duplicate charge and does not invent an amount"

Provider IDs change as vendors release models, so check the Promptfoo provider docs for current names.

Promptfoo and OpenAI

On 9 March 2026, OpenAI announced it was acquiring Promptfoo. TechCrunch reported that Promptfoo’s technology would be integrated into OpenAI Frontier, OpenAI’s enterprise platform for AI agents, and that OpenAI expected to keep building Promptfoo’s open-source offering. Promptfoo says the CLI remains open source and MIT-licensed and still supports many model providers.

If your organisation has rules about vendor ownership, review them. Evals can run locally, but some features, such as hosted red-team generation, use Promptfoo’s services. The community plan includes a monthly red-team probe allowance, and heavier use needs an enterprise plan.

DeepEval: pytest for LLM applications

DeepEval, from Confident AI, is a Python library that makes LLM testing feel like unit testing. You build a test case, pick metrics with thresholds, and a failing metric fails the pytest run.

# test_support_bot.py
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_refund_answer():
    test_case = LLMTestCase(
        input="How long do refunds take?",
        actual_output=support_bot("How long do refunds take?"),
    )
    assert_test(test_case, [AnswerRelevancyMetric(threshold=0.7)])

Here support_bot stands for your application code.

DeepEval’s metrics

DeepEval’s main strength is its metric catalogue. Its README groups them as follows:

• Custom: G-Eval, an LLM-as-a-judge metric for any criteria, and DAG, a graph-based, more deterministic judge builder.
• AI agents: Task Completion, Tool Correctness, Goal Accuracy, Step Efficiency, Plan Adherence, Plan Quality, Tool Use and Argument Correctness.
• Multi-turn: Knowledge Retention, Conversation Completeness, Turn Relevancy, Turn Faithfulness and Role Adherence.
• RAG: Answer Relevancy, Faithfulness, Contextual Recall, Contextual Precision, Contextual Relevancy and RAGAS.
• MCP: MCP Task Completion, MCP Use and Multi-Turn MCP Use.
• Voice agents: Agent Responsiveness, Turn-Taking Naturalness, Voice Naturalness, Transcription Accuracy and Audio Glitch.

DeepEval also lists integrations with frameworks including LangGraph, Pydantic AI, CrewAI, AWS AgentCore, Google ADK, Mastra, OpenAI Agents and LlamaIndex.

DeepEval and red teaming

Red teaming lives in a separate package, DeepTeam, also from Confident AI and Apache-2.0. It’s a different install from DeepEval’s metrics.

DeepEval and Confident AI

Confident AI is the hosted product from the same team, offering datasets, tracing and shared eval runs. It’s optional; DeepEval runs without an account.

Langfuse: observability and evals on real traffic

Langfuse records what an LLM application did: traces, generations, token counts, prompts, scores and datasets. Python and JavaScript SDKs send the data, and it can also ingest OpenTelemetry. Prompt management and evals sit in the same project.

What Langfuse does well

• Nested traces. You see a request that called a retriever, a model and a tool, grouped into sessions.
• Evals on stored data. Datasets, experiments, LLM-as-a-judge evaluators and human annotation queues, with scores attached to traces.
• Prompt management. Versioned, labelled prompts fetched at runtime, so production and the playground use the same text.
• Cost and token tracking. Generations can store token counts and calculated cost.
• Self-hosting. Docker Compose for local installs and Kubernetes for production.

Langfuse and ClickHouse

In January 2026, ClickHouse announced it had acquired Langfuse. Langfuse already ran on ClickHouse in both its cloud and self-hosted versions. ClickHouse’s announcement said:

• Langfuse remains open source under its existing MIT licence for core features, with self-hosting at production scale.
• Existing deployments carry on as before.
• Langfuse Cloud continues with the same SLAs and support.

Head-to-head comparison

Offline evals before release

Winner: Promptfoo or DeepEval, depending on your stack.

Promptfoo suits teams that want config-driven comparisons and a CLI. DeepEval suits Python teams who want evals in pytest. Langfuse can run experiments on datasets too, but it’s strongest when the data comes from traces.

Agent evaluation

Winner: DeepEval for metrics; Langfuse for traces.

DeepEval has the widest set of named agent metrics, including tool correctness, step efficiency and plan adherence. Langfuse gives you the traces those metrics need in production. For the overall method, see How to test AI agents in 2026.

Red teaming and security

Winner: Promptfoo.

Red teaming is built into Promptfoo’s CLI. DeepEval’s red-teaming sits in DeepTeam. Langfuse isn’t a red-teaming tool. For the threat model, see the OWASP Top 10 for Agentic Applications 2026.

Production monitoring

Winner: Langfuse.

Langfuse is built for production traffic: traces, sessions, users, costs, dashboards and online evaluation. Promptfoo and DeepEval are mainly pre-release tools, although Confident AI adds hosted tracing for DeepEval users.

CI integration

Tie between Promptfoo and DeepEval.

Promptfoo exits non-zero on failures. DeepEval failures are pytest failures. Both fit any CI system. Langfuse is usually part of CI through SDK-driven experiments rather than as the gate itself.

Language fit

• JavaScript/TypeScript teams often prefer Promptfoo, which is YAML-driven and installed via npm.
• Python teams often prefer DeepEval.
• Langfuse works with both through its SDKs.

RAG evaluation

Winner: DeepEval.

If you ship retrieval-augmented generation, DeepEval’s RAG metrics cover both halves of the pipeline. Faithfulness checks whether the answer is grounded in the retrieved context, and Contextual Precision, Recall and Relevancy check the retrieval itself. That split matters, because a wrong answer can come from bad retrieval or from a model that ignored good context, and the fixes are different.

Promptfoo can test RAG apps through HTTP targets and rubric or similarity assertions, and Langfuse can score RAG traces with judge evaluators. Neither gives you as many ready-made retrieval metrics.

Data privacy

Depends on your setup.

All three can keep data on your infrastructure. Promptfoo runs evals locally and serves results locally. DeepEval runs locally unless you send results to Confident AI. Langfuse can be fully self-hosted. The catch is the judge: model-graded metrics send inputs and outputs to whichever model provider you configure. If your test data includes customer content, either anonymise it or use a judge model you’re allowed to send it to.

Where each tool falls short

No tool is perfect. These are the limits worth knowing before you commit.

Promptfoo

• It tests the model and app behaviour you describe. It won’t click through your UI, so pair it with ordinary end-to-end tests.
• Serious eval configs become large YAML files that need review like code.
• Red-team volume on the community plan is capped, so heavy security programmes need an enterprise plan.

DeepEval

• Judge-based metrics aren’t deterministic, so thresholds can fail after a judge model change you didn’t pin.
• Every judge metric is a model call, so large datasets in CI need a budget.
• It’s Python-first. The TypeScript package exists but the Python docs are the primary reference.
• Metrics can be misused. Faithfulness without real retrieval context gives a precise-looking but meaningless number.

Langfuse

• It’s a platform to operate, not just a library. Self-hosting means running its databases and storage.
• It’s strongest with production data. Before launch, you’ll still want offline tests in another tool or in Langfuse experiments you set up yourself.
• Cloud pricing is metered on usage, so high-traffic apps should estimate volumes first.

A worked example: one bug through all three tools

Imagine a support assistant that sometimes promises refunds the policy doesn’t allow. Here’s how each tool helps.

1. Langfuse finds it. A judge evaluator scoring production traces flags answers that mention refunds. A weekly review finds five traces where the assistant promised a refund outside the 30-day window. Each trace shows the retrieved policy chunk, the prompt version and the model.
2. DeepEval pins it down. The five traces become LLMTestCase objects with the retrieved context. A Faithfulness metric fails on all five, which shows the model ignored the correct policy text rather than retrieval failing.
3. Promptfoo compares fixes. Two new prompt versions and a second model are run against the same cases plus 40 older ones. A rubric assertion checks “Does not promise a refund outside policy”, and a contains check makes sure the policy window is stated.
4. CI keeps it fixed. The winning prompt ships, and the new cases stay in the CI suite as regression tests.

The example is illustrative, not a real incident. It shows why the tools complement each other more than they compete.

Costs to plan for

All three are free to self-host or run locally, but there are real costs:

• Judge model calls. Model-graded assertions and LLM-as-a-judge metrics call another model. Large datasets in CI can cost a lot. Cache results and run full suites less often.
• Hosted plans. Promptfoo Enterprise, Confident AI and Langfuse Cloud are paid services above their free tiers. Check each vendor’s current pricing page.
• Self-hosting. Running Langfuse yourself means operating its databases and storage.
• Maintenance. Eval datasets go stale. Budget time to review and update them, just as you would test code.

Judge stability: a problem all three share

Every tool that uses LLM-as-a-judge has the same weakness: scores change when the judge model changes. To keep results trustworthy:

1. Pin the judge model version.
2. Keep a small set of human-labelled examples and check how often the judge agrees.
3. Prefer specific rubrics over vague ones.
4. Use deterministic assertions, such as exact matches, regex or schema checks, wherever possible.
5. Report pass rates over several runs, not single results.

Which should you choose?

Choose Promptfoo if:

• You want to compare prompts and models quickly.
• Red teaming is a priority.
• Your team is comfortable with YAML and a CLI.
• You work mainly in JavaScript or TypeScript.

Choose DeepEval if:

• You test in Python with pytest.
• You need a wide set of RAG, agent, multi-turn or MCP metrics.
• You want thresholds in code, reviewed in git.

Choose Langfuse if:

• You need tracing and monitoring in production.
• You want to evaluate real traffic and turn bad traces into test cases.
• You want prompt versioning alongside traces.
• You need to self-host your observability data.

Common combinations

Most mature teams use more than one tool:

• DeepEval + Langfuse: DeepEval tests in CI, Langfuse traces production, and bad traces become new DeepEval cases.
• Promptfoo + Langfuse: Promptfoo for prompt comparisons and red teaming before release, Langfuse for monitoring after release.
• Promptfoo + DeepEval: Promptfoo for security testing, DeepEval for quality metrics. This is less common, since the two overlap on offline evals.

A 30-day adoption plan

1. Week 1: Collect 30–50 real cases from logs or support tickets, with clear success criteria.
2. Week 2: Put them into Promptfoo or DeepEval, with deterministic assertions first and judge metrics second.
3. Week 3: Add tracing in staging with Langfuse or your chosen platform, and sample traces for review.
4. Week 4: Add a red-team suite and a CI gate on pass rates, and schedule a monthly dataset review.

Frequently asked questions

Is Promptfoo still open source after the OpenAI acquisition?

Promptfoo says its CLI remains open source under the MIT licence and supports multiple model providers. Some hosted features, such as red-team probe generation, have plan limits.

Is Langfuse still open source after the ClickHouse acquisition?

ClickHouse said Langfuse remains open source under its existing MIT licence for core features, with self-hosting supported at production scale.

Is DeepEval free?

Yes. DeepEval is Apache-2.0 and runs without an account. Confident AI, its hosted platform, is optional and paid above its free tier. Judge model calls are billed by your model provider.

Can I use Langfuse instead of DeepEval?

For some cases, yes. Langfuse supports datasets, experiments and LLM-as-a-judge evaluators. But if you want many ready-made metrics and pytest-style gates, DeepEval is a better fit. Many teams use both.

Which tool is best for testing AI agents?

DeepEval has the broadest named agent metrics, Langfuse gives you agent traces in production, and Promptfoo is strongest for red teaming. A combination usually works best.

Do these tools replace normal tests?

No. They test model behaviour. You still need unit, API and end-to-end tests for the software around the model.

Further reading on CheckOps

• How to test AI agents in 2026
• LLM testing and evals for vibe-coded apps
• MCP and the OWASP LLM Top 10
• Best AI testing tools 2026
• Tools: Promptfoo, DeepEval, Langfuse, OpenTelemetry