Harbor

Visit Website

Introduction

Harbor is a framework for specifying sandboxed agent tasks for evaluation and optimisation. It comes from the creators of Terminal-Bench, the benchmark for agents working in terminals, and is the official harness for Terminal-Bench 2.0.

Its scope is agents rather than single prompts. You can evaluate arbitrary agents, such as Claude Code, OpenHands and Codex CLI, against a dataset of tasks, each running in its own container. You can also build and share your own benchmarks and environments, which makes it useful for internal evals as well as public leaderboards.

Scale is a core feature. Runs launch locally with Docker, or on cloud sandbox providers with the --env flag; the README lists Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, Runta and Sail, and shows a run with 100 concurrent trials. harbor datasets list shows supported third-party benchmarks such as SWE-Bench and Aider Polyglot.

Harbor is installed as a Python tool with uv tool install harbor or pip install harbor, and every run is a CLI call naming a dataset, an agent and a model. That makes it straightforward to compare agents or models on the same tasks, or to track an internal agent’s score over time in CI.

For research use, the repository provides a citation with a Zenodo concept DOI that always resolves to the latest release.

The source code is at harbor-framework/harbor on GitHub under the Apache-2.0 licence, with about 7,000 stars; the most recent commit was on 10 November 2026.

Harbor matters because agent benchmarks are only as trustworthy as the environments they run in. Containerised, reproducible tasks mean two teams can run the same evaluation and compare results honestly, and the same harness can be pointed at your own tasks rather than public benchmarks.

For product teams, the practical use is regression testing for agents. Package a set of realistic tasks from your domain, run them against each new model or prompt version, and track pass rates over time. A drop on a task you care about is a far more useful signal than a leaderboard number.

Expect a learning curve. Building good tasks, with clear success criteria and isolated environments, takes effort, but that work is reusable across every future model upgrade.

There is also a governance benefit. When a model provider ships a new version, a ready-made Harbor task suite lets you answer a concrete question within hours: does the new model do our work better, worse or the same? That turns model upgrades from a leap of faith into a measured decision, and it creates an audit trail of why a particular model was chosen.

Keep tasks honest. Avoid tasks the model may have seen in public training data, refresh them periodically, and record the exact environment, model and prompt versions with each run so results can be reproduced later.

Start with ten to twenty well-designed tasks rather than hundreds of weak ones. A small, trusted suite that runs quickly on every model or prompt change is more useful than a huge benchmark that is too slow and expensive to run often.

Features

1. Agent evaluation: Claude Code, OpenHands, Codex CLI and more.
2. Sandboxed tasks: Each task runs in its own container.
3. Custom benchmarks: Build and share environments.
4. Terminal-Bench 2.0: Official harness.
5. Third-party datasets: SWE-Bench, Aider Polyglot and others.
6. Cloud parallelism: Daytona, Modal and other sandbox providers.
7. Concurrency control: --n-concurrent sets parallel trials.
8. CLI-first: One command per experiment.

Pros

1. Realistic agent tests: Agents act in real environments, not chat transcripts.
2. Scales out: Thousands of environments in parallel.
3. Free to use: Open source under the Apache-2.0 licence, so the core tool costs nothing to use.
4. Actively maintained: The repository had commits as recent as 10 November 2026.
5. Community: About 7,000 GitHub stars.

Cons

1. Infrastructure costs: Cloud sandboxes and model calls add up.
2. Benchmark-oriented: Less suited to production monitoring.
3. Docker needed locally: Local runs require Docker.

Usage and Links (Example)

uv tool install harbor
export ANTHROPIC_API_KEY=<key>
harbor run --dataset terminal-bench@2.0 --agent claude-code \
  --model anthropic/claude-opus-5-5 --n-concurrent 4
harbor datasets list
harbor run -d "<dataset@version>" -m "<model>" -a "<agent>"

Resources:

• Harbor website
• GitHub repository

Pricing

Harbor is free and open source under the Apache-2.0 licence.

Recommended for?

1. Teams building coding or terminal agents who need repeatable benchmarks.
2. Researchers comparing agents and models.
3. Platform teams choosing which agent to adopt.

Conclusion

Harbor is the most direct way to run Terminal-Bench-style agent evaluations, and it generalises to your own tasks.

Use it to compare agents on your own repository’s tasks before rolling one out.

On CheckOps, Harbor is filed under the Release & CI gates and Integration testing stages. Related tools listed here include Inspect, LM Evaluation Harness, OpenAI Evals, Promptfoo, Braintrust and DeepEval. Which one fits best depends mostly on your stack, your team’s size and skills, and whether you prefer to run tools yourself or pay for a managed service. This page reflects the official website, documentation and repository as they stood on 11 November 2026.