Pydantic Evals

Visit Website

Introduction

Pydantic Evals is a code-first evaluation framework that ships with the Pydantic AI project, the Python agent framework from the team behind Pydantic. It lets you define datasets of cases with inputs and expected outputs, run a task function over them, and score results with evaluators.

Evaluators can be deterministic checks, type checks or LLM-as-judge rubrics, and datasets can be written in Python or YAML with schemas generated from types. Reports print in the terminal and can be sent to Pydantic Logfire through OpenTelemetry for comparison over time. It works for any Python function, not only Pydantic AI agents.

In the quality loop it is a lightweight regression suite for LLM features.

The source code is at pydantic/pydantic-ai on GitHub under the MIT licence, with about 21,100 stars; the most recent commit was on 11 November 2026.

Features

1. Typed datasets: Cases with typed inputs and outputs.
2. Evaluators: Deterministic and LLM-judge.
3. YAML or Python: Flexible dataset formats.
4. Span-based evaluation: Assert on traced behaviour.
5. Reports: Terminal and Logfire views.
6. Framework-agnostic: Evaluate any Python function.

Pros

1. Fits Python developers: Feels like writing tests.
2. Free to use: Open source under the MIT licence, so the core tool costs nothing to use.
3. Actively maintained: The repository had commits as recent as 11 November 2026.
4. Community: About 21,100 GitHub stars.

Cons

1. Python only: No other languages.
2. UI via Logfire: Rich history needs Logfire.
3. Young API: Still evolving.

Usage and Links (Example)

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import IsInstance

dataset = Dataset(cases=[Case(name="capital", inputs="What is the capital of France?", expected_output="Paris")],
                  evaluators=[IsInstance(type_name="str")])
report = dataset.evaluate_sync(answer_question)
report.print()

Resources:

• Pydantic Evals website
• GitHub repository

Pricing

Pydantic Evals is free and open source under the MIT licence.

Recommended for?

1. Pydantic AI users.
2. Python teams wanting code-first LLM tests.

Conclusion

Pydantic Evals brings a familiar, typed testing style to LLM evaluation.

Pair it with Logfire if you want history and dashboards.

On CheckOps, Pydantic Evals is filed under the Release & CI gates stage. Related tools listed here include DeepEval, Promptfoo, Ragas, Inspect, Braintrust and LangSmith. When comparing them, weigh the licence, the languages and platforms each supports, whether it is self-hosted or managed, and how cleanly it slots into your existing CI pipeline. Details on this page were checked against the official website, documentation and repository on 11 November 2026.