AgentEvals

Visit Website

Introduction

AgentEvals focuses on agent trajectory, the sequence of intermediate steps an agent takes while solving a task. Because agents control their own flow, a change in one place can ripple downstream, and checking only the final answer misses that.

The package offers trajectory evaluators, including LLM-as-a-judge evaluators (the quickstart uses OpenAI’s o3-mini) and match-based comparisons against a reference trajectory. For general LLM output evaluation, LangChain points to its companion package openevals.

The source code is at langchain-ai/agentevals on GitHub under the MIT licence, with about 759 stars; the most recent commit was on 14 July 2026.

Features

1. Trajectory match: Compare steps to a reference.
2. LLM-as-judge: Grade trajectories with a model.
3. Python and JS: Packages for both.
4. Framework-agnostic: Takes OpenAI-style message lists.
5. Companion to openevals: General evaluators live there.

Pros

1. Lightweight: Drop into existing tests.
2. Free to use: Open source under the MIT licence, so the core tool costs nothing to use.
3. Actively maintained: The repository had commits as recent as 14 July 2026.

Cons

1. Narrow scope: Trajectories only.
2. Judge cost: LLM judges need API keys.

Usage and Links (Example)

pip install agentevals
export OPENAI_API_KEY=<key>

Use a trajectory evaluator in pytest or in LangSmith experiments.

Resources:

• AgentEvals website

Pricing

AgentEvals is free and open source under the MIT licence.

Recommended for?

1. Teams testing tool-using agents where the path matters as much as the answer.

Conclusion

AgentEvals is a small, focused library for checking how an agent reached its answer.

Combine it with outcome evals for full coverage.

On CheckOps, AgentEvals is filed under the Release & CI gates and Integration testing stages. Related tools listed here include OpenEvals, LangSmith, DeepEval, Pydantic Evals, Inspect and Promptfoo. Which one fits best depends mostly on your stack, your team’s size and skills, and whether you prefer to run tools yourself or pay for a managed service. This page reflects the official website, documentation and repository as they stood on 11 November 2026.