CheckOps home

Tools / Best tools / LLM evaluation

CheckOps

Best LLM evaluation tools in 2026

These are LLM and agent evaluation tools in the CheckOps directory, including entries tagged for LLM eval or LLM and agent evaluation. The directory has 62 tagged entries in this area; this page lists 15.

How we picked

CheckOps is a genuine and free directory. There are no paid sponsors, featured listings, or paid placement on these pages. The shortlist is built from tools already in the directory whose tags match the topic, with a preference for entries that have a fuller CheckOps page so the facts in the table can be checked against the source essay. This is not a claim that one tool is objectively better than another, and it is not a paid ranking.

Well-known tools that already have a CheckOps page in this area are listed first. The rest of the shortlist is filled from other tagged entries, preferring fuller essays so the facts in the table can be checked. That order is a way to keep the page useful, not a quality score and not a paid ranking.

Tools

  1. P

    1. Promptfoo

    Promptfoo is an open-source CLI for LLM evals and red teaming. The community tool is free; enterprise features are quoted.

    Read the CheckOps page · Open source

  2. D

    2. DeepEval

    DeepEval is an open-source LLM evaluation library with pytest-style asserts. Confident AI is the optional paid cloud from the same team.

    Read the CheckOps page · Open source

  3. R

    3. Ragas

    Ragas is an open-source library for systematically evaluating LLM applications, including RAG systems and agents.

    Read the CheckOps page · Open source

  4. L

    4. LangSmith

    LangSmith is LangChain's platform for observability and evaluation of LLM applications and agents, including offline and online evals.

    Read the CheckOps page · Commercial, with a free or trial tier tagged

  5. N

    5. NeMo Guardrails

    NeMo Guardrails is an Apache-2.0 toolkit from NVIDIA for adding programmable guardrails to LLM-based conversational systems, using Colang flows and input, output, dialog and retrieval rails.

    Read the CheckOps page · Open source

  6. G

    6. Guardrails AI

    Guardrails AI is an Apache-2.0 Python framework for adding input and output validation to LLM applications, with a hub of validators and structured output generation.

    Read the CheckOps page · Open source

  7. C

    7. Confident AI

    Confident AI is the hosted LLM evaluation and observability platform from the makers of DeepEval and DeepTeam, covering evals in CI, tracing, voice agents, conversation simulation, red teaming and AI governance.

    Read the CheckOps page · Commercial, with a free or trial tier tagged

  8. O

    8. OpenLIT

    OpenLIT is an Apache-2.0, OpenTelemetry-native platform for AI agents and coding agents that combines LLM tracing, evals with CI gates, guardrails, prompt management, cost tracking and GPU monitoring, free to self-host.

    Read the CheckOps page · Open source

  9. L

    9. Lakera

    Lakera, now part of Check Point, offers AI security products: a Guard API (documented as Check Point AI Guardrails) for runtime screening of prompt injection and data leaks, and AI red teaming.

    Read the CheckOps page · Commercial, with a free or trial tier tagged

  10. A

    10. Arthur Engine

    Arthur Engine is an MIT open-source service from Arthur for monitoring and governing AI/ML workloads, with real-time guardrails for PII, hallucination, prompt injection and toxicity.

    Read the CheckOps page · Open source

  11. L

    11. Langfuse

    Langfuse is an open-source platform for LLM traces, prompts, and evals. Self-hosting is free; Langfuse Cloud is paid after a hobby tier.

    Read the CheckOps page · Open source, with commercial options tagged

  12. F

    12. Fiddler AI

    Fiddler AI is a commercial AI observability and governance platform for agents, LLMs and ML models, with guardrails, evaluations and monitoring; CrowdStrike has announced it will acquire Fiddler.

    Read the CheckOps page · Commercial, with a free or trial tier tagged

  13. W

    13. W&B Weave

    W&B Weave is an Apache-2.0 toolkit from Weights & Biases for tracing and evaluating AI agents and LLM apps, with scorers, datasets, evaluations and an online UI.

    Read the CheckOps page · Open source

  14. M

    14. MLflow GenAI Evaluation

    MLflow is an Apache-2.0 open-source AI engineering platform whose GenAI features provide tracing, LLM-judge scorers, evaluation datasets and production monitoring for agents and LLM apps.

    Read the CheckOps page · Open source

  15. E

    15. Evidently

    Evidently is an Apache-2.0 open-source ML and LLM evaluation and observability framework with 100+ built-in metrics, test suites, reports and a monitoring UI.

    Read the CheckOps page · Open source

Comparison

License, pricing model, and languages or platforms are taken from tags on each CheckOps page. Pricing notes are a short excerpt from that page's Pricing section when one exists. A blank or "not stated" cell means the CheckOps page does not record it.

Comparison of Best LLM evaluation tools in 2026
ToolLicensePricing modelLanguages / platformsPricing notes
PromptfooOpen sourceFree, Open sourceNot stated in the tags on the CheckOps pageFrom promptfoo.dev/pricing: Community: Free. All LLM evaluation features, all model providers and integrations, red teaming up to 10,000 probes per month, local or self-hosted runs, vulnerability scanning, and community…
DeepEvalOpen sourceFree, Open sourceNot stated in the tags on the CheckOps pageDeepEval: Free, Apache-2.0. No account is required to assert metrics locally or in CI. Judge-model tokens are paid to the model provider. Confident AI Free: $0.
RagasOpen sourceFree, Open sourceNot stated in the tags on the CheckOps pageFree and open source under Apache-2.0. LLM judge calls are billed by your model provider.
LangSmithCommercial, with a free or trial tier taggedFree tier, Paid, SaaSNot stated in the tags on the CheckOps pageAt the time of review LangChain listed a Developer plan at $0 per seat (one seat, then pay as you go, with a monthly trace allowance), Plus at $39 per seat per month with pay-as-you-go usage, and custom Enterprise…
NeMo GuardrailsOpen sourceFree, Open sourcePythonNeMo Guardrails is free and open source under the Apache-2.0 licence.
Guardrails AIOpen sourceFree, Open sourcePythonGuardrails AI is free and open source under the Apache-2.0 licence.
Confident AICommercial, with a free or trial tier taggedFree tier, Paid, SaaSNot stated in the tags on the CheckOps pageConfident AI lists a forever-free plan, a $200 a month plan and a $2,000 a month plan, each with tracing billed at $1 per GB-month ingested or retained; enterprise options are custom.
OpenLITOpen sourceFree, Open sourcePythonOpenLIT is free and open source under the Apache-2.0 licence.
LakeraCommercial, with a free or trial tier taggedFree tier, Paid, SaaSNot stated in the tags on the CheckOps pageLakera offers a free way to start (Start for free on its site) and enterprise plans through a demo with sales.
Arthur EngineOpen sourceFree, Open sourceNot stated in the tags on the CheckOps pageArthur Engine is free and open source under the MIT licence.
LangfuseOpen source, with commercial options taggedFree, Open source, SaaSNot stated in the tags on the CheckOps pageFrom langfuse.com/pricing. Cloud units are traces, observations, and scores. Self-host: Free under the MIT license. You pay for machines and operations. Hobby: $0.
Fiddler AICommercial, with a free or trial tier taggedFree tier, Paid, SaaSNot stated in the tags on the CheckOps pageFiddler's pricing page lists a Free plan, usage pricing of $0.002 per trace for the next tier, and enterprise plans through sales.
W&B WeaveOpen sourceFree, Open sourcePythonThe Weave SDK is open source under Apache-2.0. W&B's pricing page lists a Free plan at $0 per month and paid plans starting at $60 per month billed monthly, with a 30-day free trial; academic research use is free.
MLflow GenAI EvaluationOpen sourceFree, Open sourcePythonMLflow GenAI Evaluation is free and open source under the Apache-2.0 licence.
EvidentlyOpen sourceFree, Open sourcePythonEvidently's open-source library is free under Apache-2.0. Evidently Cloud, the hosted platform, offers free and paid plans; see evidentlyai.com for current pricing.

FAQ

How did CheckOps pick these tools?
CheckOps is a genuine and free directory. There are no paid sponsors, featured listings, or paid placement on these pages. The shortlist is built from tools already in the directory whose tags match the topic, with a preference for entries that have a fuller CheckOps page so the facts in the table can be checked against the source essay. This is not a claim that one tool is objectively better than another, and it is not a paid ranking.
Does CheckOps sell placement on this list?
No. CheckOps does not sell featured or paid placement, and this page does not include sponsored listings.
How many llm evaluation tools are in the directory?
This shortlist has 15 tools. The directory has 62 tagged entries in this area. See the LLM & agent evaluation category for the wider set.
Are the one-line summaries complete reviews?
No. Each line is taken from the tool's CheckOps page (the short description, or the opening of the essay when that description is empty). Open the linked page for the full write-up.