TestDriver

Visit Website

Introduction

TestDriver describes itself as an “AI code reviewer that actually runs your app”. According to the official site, it runs every pull request in a real desktop sandbox, finds bugs and builds regression tests, posting results back on GitHub.

Its SDK, published at testdriverai/testdriverai, is a plugin for Vitest that spawns ephemeral devices and uses vision-based LLMs to build tests. The README lists targets that are awkward for DOM-based tools: third-party web apps, desktop apps, VS Code and Chrome extensions, OAuth flows, PDF content, file uploads and OS accessibility.

Features

1. PR review by running the app: Each pull request is exercised in a sandbox and reported as a GitHub check.
2. Vision-based actions: Tests find elements by description, for example testdriver.find("Recycle Bin icon"), and assert in natural language.
3. Vitest plugin: The SDK runs inside Vitest, so tests live in your repository as code.
4. Desktop and extension coverage: Examples include Java Swing and Qt apps, Chrome extensions and VS Code.
5. Regression test generation: The bot can open a PR that adds an end-to-end test, as shown on the homepage.
6. Playwright MCP and screenshot testing: Both are listed as platform capabilities.
7. Self-hosted enterprise: An Enterprise self-hosted option offers unlimited testing without per-minute fees.

Pros

1. Reaches beyond the browser: Desktop apps, extensions and OS dialogs are in scope.
2. Tests as code: Generated tests are Vitest files you can review and version.
3. Transparent metering: Per-seat price plus a published per-minute rate.
4. Trial without charge: 14 days or 360 testing minutes, whichever comes first.
5. GitHub-native: Runs and results appear on pull requests.

Cons

1. Per-seat billing on GitHub users: Seats are billed for every GitHub user who gets a review or runs a test, which can grow quickly.
2. Minute overages: Testing beyond the included minutes costs $0.08 a minute.
3. No licence file in the SDK repo: At review time GitHub showed no detected licence for the repository; check before reusing the code.
4. Vision costs time: Vision-model steps are slower than DOM selectors.
5. Card required for trial: A card is added at checkout, though not charged during the trial.

Usage and Links (Example)

Install the GitHub app from the site, then let TestDriver review pull requests. For code-first tests, the README shows calls like:

await testdriver.find("New text document").mouseDown();
await testdriver.find("Recycle Bin icon").mouseUp();
const result = await testdriver.assert('the "New Text Document" icon is not visible on the Desktop');
expect(result).toBeTruthy();

See the docs for Vitest setup and sandbox options.

Resources:

• TestDriver website
• Pricing
• Docs
• GitHub repository

Pricing

From TestDriver’s pricing page:

1. Pro Cloud: $24 a month per seat, plus $0.08 per testing minute beyond the included 360 minutes per billing period. Public and private repositories, unlimited code reviews. A 14-day free trial (or 360 minutes) applies.
2. Enterprise Self-Hosted: Custom. Unlimited testing with no per-minute fees, shared licences, expert setup.

Recommended for?

1. Teams shipping desktop apps, extensions or complex OAuth flows that DOM tools struggle with.
2. GitHub-centred teams wanting an AI reviewer that runs the app on each PR.
3. Engineers who want generated tests as code in Vitest.
4. Not a fit if seat-per-GitHub-user billing does not suit a large contributor base.

Conclusion

TestDriver applies vision-based AI to pull-request testing and reaches surfaces that selector-based frameworks rarely cover, such as desktop apps and browser extensions.

Pricing is clear but can scale with contributors and minutes. Check the repository licence and try the trial on your hardest-to-test flow.