Promptfoo

Visit Website

Introduction

Promptfoo evaluates LLM prompts and applications from the command line. You declare providers (the models), prompts or a target app, and assertions. promptfoo eval runs the matrix and writes a table of pass and fail. promptfoo redteam generates adversarial cases for prompt injection, jailbreaks, and similar risks. Results can be viewed in a local web UI. The project is MIT-licensed at promptfoo/promptfoo.

Promptfoo’s own site states that the company is now part of OpenAI, that the CLI stays open source and MIT-licensed, and that it still targets many model providers, not only OpenAI. Enterprise features (shared dashboards, SSO, higher red-team limits) are a commercial plan. Model API calls are billed by the model provider either way. This is an eval tool, not a functional UI runner.

Features

1. Eval configs: A YAML file lists prompts, providers, and tests. A test is an input plus assertions: equals, contains, regex, similarity, or a model-graded rubric.
2. Many providers: OpenAI, Anthropic, Azure, Bedrock, Ollama, and HTTP endpoints. Comparing two models is a second provider entry, not a second tool.
3. Local UI: promptfoo view serves the latest eval so you can diff outputs without uploading them.
4. CI gate: The eval exits non-zero when assertions fail, so a pull request can block on a prompt change.
5. Red teaming: promptfoo redteam run generates probes against a target. The community plan includes a monthly probe allowance for that hosted capability.
6. Caching: Repeat evals can reuse outputs, which matters because each live call costs money and time.
7. Application targets: You can point tests at your own HTTP app, not only at a raw model endpoint, so the eval covers the retrieval and tools around the model.
8. Reports: HTML and JSON output for a single run. Team sharing and continuous monitoring are enterprise features, not the default CLI.

Pros

1. Runs locally: Prompts and outputs stay on the machine unless you opt into a hosted feature. That matters for unreleased product data.
2. Assertions are explicit: A rubric or an exact match is in the YAML, so a “better” prompt has to pass the same tests.
3. Provider-agnostic: The config can compare models. An OpenAI acquisition does not, on Promptfoo’s current statement, remove other providers from the OSS CLI.
4. Red teaming is built in: You do not need a second project to start adversarial tests, within the community probe limit.
5. Free to start: The community plan is “free forever” on promptfoo.dev/pricing, including eval features and local or self-hosted runs.

Cons

1. Judge costs and judge quality: Model-graded assertions call another model. The score moves when the judge model changes, and you pay for those calls.
2. Red-team allowance: The community plan caps red-team probes at 10,000 per month. Heavier security programs need an enterprise quote.
3. Not a product test suite: Promptfoo will not click your settings page. It tests model and app behaviour you described. Pair it with ordinary tests for the software around the model.
4. Config size: A serious eval is a large YAML of cases. It needs review like test code, or it becomes a stale demo.
5. Ownership change: Promptfoo has said it is joining OpenAI and that the CLI remains MIT and multi-provider. Re-read that commitment if your policy forbids sending evals through an OpenAI-owned tool, even locally.

Usage and Links (Example)

npm install -g promptfoo
promptfoo init
promptfoo eval
promptfoo view

A minimal promptfooconfig.yaml:

prompts:
  - "Summarize this in one sentence: {{text}}"
providers:
  - openai:gpt-4.1-mini
tests:
  - vars:
      text: "Promptfoo runs evals from a CLI."
    assert:
      - type: contains
        value: CLI

Set the provider’s API key in the environment before promptfoo eval. Use promptfoo eval --no-cache when you need a clean measurement. For red teaming, promptfoo redteam setup writes a config; read it before you run probes at a production URL.

Resources:

• Promptfoo website
• Getting started
• Pricing
• GitHub: promptfoo/promptfoo

Pricing

From promptfoo.dev/pricing:

1. Community: Free. All LLM evaluation features, all model providers and integrations, red teaming up to 10,000 probes per month, local or self-hosted runs, vulnerability scanning, and community support. You still pay the model provider for tokens.
2. Enterprise: Custom price. Higher red-team limits, team sharing, continuous monitoring, a security dashboard, SSO, API access, managed cloud, and an SLA.
3. On-premise: Custom price. Enterprise features deployed in your infrastructure, with a dedicated runner.

There is no published dollar figure for Enterprise. Probe overage beyond the community 10,000 is an enterprise conversation, not a self-serve meter on the pricing page.

Recommended for?

1. Engineers changing prompts or models who want a repeatable pass/fail table before they ship.
2. Security reviews of an LLM feature that can start with the built-in red-team probes and a local target.
3. CI on a prompt repo, where the eval config is the test suite.
4. Teams that must keep prompts on-machine. Use the CLI locally. Move to Enterprise only when you need shared monitoring and SSO.

Conclusion

Promptfoo is the CLI to reach for when an LLM change needs assertions and a diff, not another manual chat session. Keep the community edition for local evals, watch the 10,000-probe red-team cap, and budget model tokens separately. Enterprise has no public price. The tool does not replace end-to-end tests of the application around the model.