CheckOps home

Tools directory

I

CheckOps

Inspect

Inspect is the UK AI Security Institute's MIT-licensed framework for LLM evaluations, with datasets, solvers, scorers, agents, sandboxed tools and 200+ pre-built evals.

Visit website

Latest public signals

Not refreshed yet. Refresh reads the public site and, when the entry has a GitHub repository, the latest release.

Visit Website

Introduction

Inspect is a framework for large language model evaluations created by the UK AI Security Institute. The repository UKGovernmentBEIS/inspect_ai is MIT-licensed.

It provides built-in components for prompt engineering, tool usage, multi-turn dialogue and model-graded evaluations. A separate collection offers over 200 pre-built evaluations ready to run on any model. The docs cover tasks, datasets, solvers, scorers, agents (ReAct, deep agents, multi-agent), tools including MCP and sandboxing, and a log viewer with a VS Code extension.

Features

  1. Tasks, solvers, scorers: Composable building blocks for evals.
  2. Agents: ReAct and custom agents, multi-agent setups and an agent bridge.
  3. Tools and sandboxing: Bash, Python, web search, computer use, MCP tools and tool approval.
  4. Model providers: Many providers, caching, batch mode and fallbacks.
  5. Scoring: Standard and custom scorers, model grading and metrics.
  6. Log viewer: Browse results in a viewer or VS Code.
  7. 200+ evals: Pre-built evaluations in a companion collection.

Pros

  1. Government-backed and open: Built by a safety institute, MIT-licensed.
  2. Agent-ready: Strong support for tool-using agents in sandboxes.
  3. Extensible: Extensions via Python packages.
  4. Good docs: Comprehensive user guide, also published for coding agents.
  5. Active: Commits on the day of review.

Cons

  1. Research orientation: Designed for model evaluation; app-level testing needs adaptation.
  2. Python only: Python framework.
  3. Costs: Running large evals costs model tokens.
  4. Learning curve: Many concepts to learn.
  5. No hosted UI: Self-run tooling.

Install with pip (pip install inspect-ai), write a task with a dataset, solver and scorer, then run it from the command line, for example inspect eval your_task.py --model <provider/model>. Open the log viewer to inspect samples. See the tutorial at inspect.aisi.org.uk.

Resources:

Pricing

Inspect is free and open source under MIT.

  1. Teams evaluating models or agents with tools in sandboxes.
  2. Safety and security researchers.
  3. Engineers wanting a rigorous Python eval framework.
  4. Not a fit for no-code users.

Conclusion

Inspect is a rigorous, open-source eval framework with strong agent and sandbox support from the UK AI Security Institute.

It suits teams willing to write Python evals and run them themselves.

Alternatives

Other tools on CheckOps that cover similar ground. Open a page, or compare one with Inspect side by side.