CheckOps home

Tools directory

A

CheckOps

AgentOps

AgentOps is an MIT Python SDK and app for monitoring AI agents, tracking LLM calls, costs, tool use and sessions, with replays for debugging agent runs.

Visit website

Latest public signals

Not refreshed yet. Refresh reads the public site and, when the entry has a GitHub repository, the latest release.

Visit Website

Introduction

AgentOps helps developers build, evaluate and monitor AI agents, from prototype to production. The README says initialising the client gives analytics on all LLM calls, and the app is open source under the MIT licence.

It records sessions with LLM calls, tool invocations, errors, costs and latency, and provides session replays to step through what an agent did. Integrations cover many agent frameworks, such as CrewAI, AutoGen, LangChain and the OpenAI Agents SDK. The full app can be run locally from the repository.

In the quality loop AgentOps helps debug and monitor agent behaviour.

The source code is at AgentOps-AI/agentops on GitHub under the MIT licence, with about 6,000 stars; the most recent commit was on 25 June 2026.

Features

  1. Session tracking: Calls, tools and errors per run.
  2. Cost tracking: Token and spend analytics.
  3. Replays: Step through agent sessions.
  4. Framework integrations: Popular agent frameworks.
  5. Decorators: Mark agents, tools and operations.
  6. Self-hostable app: Dashboard and API in the repo.

Pros

  1. Agent-centric views: Built around sessions and tools.
  2. Free to use: Open source under the MIT licence, so the core tool costs nothing to use.
  3. Actively maintained: The repository had commits as recent as 25 June 2026.
  4. Community: About 6,000 GitHub stars.

Cons

  1. Slower releases: Last tagged release was August 2025.
  2. Python focus: SDK is mainly Python.
  3. Evaluation depth: Monitoring more than scoring.
import agentops
agentops.init(api_key="...")
# run your agent; LLM calls are recorded in a session

Resources:

Pricing

The AgentOps SDK and app are open source under MIT. A hosted service is available; see agentops.ai for plans.

  1. Python agent developers needing quick visibility.

Conclusion

AgentOps gives a simple window into what agents do and cost.

Check integration support for your framework version.

On CheckOps, AgentOps is filed under the Production monitoring & observability and Release & CI gates stages. Related tools listed here include Langfuse, Laminar, Helicone, OpenLLMetry, Arize Phoenix and Opik. A short trial on a real project, with your own code, data and pipeline, is usually the fastest way to choose between them, because fit matters more than feature lists. Facts here come from the official site, documentation and repository, checked on 11 November 2026.

Alternatives

Other tools on CheckOps that cover similar ground. Open a page, or compare one with AgentOps side by side.