MLflow GenAI Evaluation

Visit Website

Introduction

MLflow describes itself as the largest open-source AI engineering platform for agents, LLMs and ML models. Long known for experiment tracking and model registry, MLflow 3 added a GenAI stack for tracing, evaluating and monitoring LLM applications, and this page covers those evaluation features.

Tracing is OpenTelemetry-compatible and can be enabled automatically for many libraries, including OpenAI, Anthropic, LangChain, LlamaIndex, DSPy and others, or manually with decorators. Traces capture prompts, tool calls, retrieval steps, latencies and token usage and are viewed in the MLflow UI.

Evaluation uses mlflow.genai.evaluate() with a dataset and a list of scorers. Built-in LLM-judge scorers check qualities such as correctness, relevance to the query, groundedness in retrieved context, safety and adherence to guidelines written in plain language, and custom scorers can be any Python function. Evaluation datasets can be built from production traces and versioned, and results are logged as runs so different prompt or model versions can be compared side by side.

The same scorers can run on production traffic for monitoring, and human feedback can be collected through review apps and attached to traces. Prompt management versions prompts alongside code. Because it is part of MLflow, GenAI evaluation sits next to classic experiment tracking, so teams with ML and LLM workloads use one platform. MLflow can be self-hosted or used as a managed service from several cloud providers, including Databricks.

In the quality loop MLflow GenAI evaluation is a release gate for prompts, models and agent logic, and a production monitor once changes are live.

The source code is at mlflow/mlflow on GitHub under the Apache-2.0 licence, with about 28,800 stars; the most recent commit was on 11 November 2026.

Teams already using MLflow for classic models get the most from it by reusing the same tracking server for GenAI work, so prompts, traces, evaluation runs and model versions share one history. Define a small set of scorers that match the product’s risks, such as correctness, groundedness and safety, run them on a fixed evaluation dataset in CI, and compare each run with the previous release before promoting a new prompt or model.

Tracing in production closes the loop: traces from real traffic can be reviewed, labelled and added to the evaluation dataset, so the test set grows with the failure modes users actually hit.

Keep evaluation datasets versioned so score changes can be traced to data or code.

Features

1. Automatic tracing: OpenTelemetry-compatible traces for many libraries.
2. mlflow.genai.evaluate: Run datasets through scorers.
3. LLM-judge scorers: Correctness, relevance, groundedness, safety and guidelines.
4. Custom scorers: Python functions as metrics.
5. Evaluation datasets: Versioned datasets, including from traces.
6. Run comparison: Compare versions in the UI.
7. Production monitoring: Scorers on live traces.
8. Human feedback: Review apps and annotations.
9. Prompt registry: Version and manage prompts.
10. Self-hosted or managed: Run anywhere, or use a cloud-managed service.

Pros

1. One platform: LLM evaluation alongside ML tracking and registry.
2. Open source and widely adopted: Large community and ecosystem.
3. Free to use: Open source under the Apache-2.0 licence, so the core tool costs nothing to use.
4. Actively maintained: The repository had commits as recent as 11 November 2026.
5. Community: About 28,800 GitHub stars.

Cons

1. Heavyweight: More platform than a lightweight eval library.
2. Version differences: GenAI APIs changed significantly in MLflow 3.
3. Judge calibration: LLM judges need checking against human labels.
4. Some features managed-only: Certain capabilities are richer in managed offerings.

Usage and Links (Example)

pip install "mlflow>=3"
mlflow server --port 5000

import mlflow
from mlflow.genai.scorers import Correctness, RelevanceToQuery, Guidelines

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.openai.autolog()

def predict_fn(question: str) -> str:
    return my_agent(question)

data = [{"inputs": {"question": "What is our refund window?"},
         "expectations": {"expected_response": "30 days"}}]

mlflow.genai.evaluate(
    data=data, predict_fn=predict_fn,
    scorers=[Correctness(), RelevanceToQuery(),
             Guidelines(name="tone", guidelines="Answer politely and concisely")],
)

Practical tips:

1. Start with autolog tracing, then build an evaluation dataset from real traces.
2. Express product rules as Guidelines scorers that non-engineers can read.
3. Reuse the same scorers for production monitoring.

Resources:

• MLflow GenAI Evaluation website
• GitHub repository

Pricing

MLflow GenAI Evaluation is free and open source under the Apache-2.0 licence.

Recommended for?

1. Teams already using MLflow for ML experiments.
2. Organisations wanting a self-hosted, open-source LLM evaluation stack.
3. Agent builders who need tracing, evaluation and monitoring together.
4. Smaller projects may prefer a lighter library such as DeepEval or promptfoo.

Conclusion

MLflow’s GenAI features turn a widely used ML platform into a full LLM quality stack: traces, judges, datasets, comparisons and monitoring in one open-source tool.

It suits teams that want a single platform for ML and LLM work; calibrate judges early and keep evaluation datasets versioned with your code.

On CheckOps, MLflow GenAI Evaluation is filed under the Release & CI gates and Production monitoring & observability stages. Related tools listed here include DeepEval, Promptfoo, Ragas, Inspect, OpenAI Evals and Braintrust. Which one fits best depends mostly on your stack, your team’s size and skills, and whether you prefer to run tools yourself or pay for a managed service. This page reflects the official website, documentation and repository as they stood on 11 November 2026.