OpenEvals

Visit Website

Introduction

OpenEvals provides readymade evaluators for LLM applications. Its README compares evals to tests in traditional software and positions the package as a starting point from which you write more application-specific evals; agent-specific evaluators live in the sibling agentevals package.

Evaluators include LLM-as-judge with prebuilt prompts for correctness, conciseness, hallucination and more, structured-output matching, exact match, Levenshtein and embedding similarity, and code evaluators for type checking and execution. It ships for Python and TypeScript, works standalone, and integrates with LangSmith for logging.

In the quality loop it supplies assertions for LLM test suites.

The source code is at langchain-ai/openevals on GitHub under the MIT licence, with about 1,200 stars; the most recent commit was on 2 November 2026.

Features

1. LLM-as-judge: Prebuilt judge prompts.
2. String and embedding metrics: Exact, edit distance, similarity.
3. Structured output: Compare JSON outputs.
4. Code evaluators: Type check and run code.
5. Python and TypeScript: Two SDKs.
6. LangSmith integration: Optional logging.

Pros

1. Framework-agnostic: Works without LangChain.
2. Free to use: Open source under the MIT licence, so the core tool costs nothing to use.
3. Actively maintained: The repository had commits as recent as 2 November 2026.

Cons

1. Building blocks only: No runner or UI of its own.
2. Judge cost: LLM judges cost money.
3. Young: Version 0.x.

Usage and Links (Example)

from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT
judge = create_llm_as_judge(prompt=CORRECTNESS_PROMPT, model="openai:o3-mini")
print(judge(inputs=q, outputs=answer, reference_outputs=expected))

Resources:

• OpenEvals website

Pricing

OpenEvals is free and open source under the MIT licence.

Recommended for?

1. Developers writing LLM tests in pytest or Vitest.

Conclusion

OpenEvals gives a quick set of reusable evaluators.

Calibrate judge prompts against human labels.

On CheckOps, OpenEvals is filed under the Release & CI gates stage. Related tools listed here include DeepEval, Promptfoo, Ragas, Inspect, Braintrust and LangSmith. A short trial on a real project, with your own code, data and pipeline, is usually the fastest way to choose between them, because fit matters more than feature lists. Facts here come from the official site, documentation and repository, checked on 11 November 2026.