Midscene.js

Visit Website

Introduction

Midscene.js is an open-source GUI agent and end-to-end testing framework from the web-infra-dev organisation on GitHub. It is MIT-licensed. The official site’s tagline is to “see interfaces like a human” and write E2E tests in natural language.

It covers web (Playwright, Puppeteer and a Chrome bridge), Android, iOS, HarmonyOS and desktop. Tests can be written as YAML with steps like aiAct and aiWaitFor, or in JavaScript integrated with Playwright Test, Rstest or Vitest. It works with vision-capable models you configure.

Features

1. Natural-language steps: aiAct, aiWaitFor and related calls describe what a user does and expects.
2. YAML test framework: @midscene/test runs YAML cases with built-in nodes across platforms.
3. JavaScript SDK: Integrations for Playwright Test, Rstest and Vitest, plus standalone scripts.
4. Many platforms: Web, Chrome bridge, Android, iOS, HarmonyOS and desktop.
5. Model choice: Guides cover model selection and configuration, including an API key or a ChatGPT subscription.
6. Reports: Visual reports help debug what the agent saw and did.
7. Agent docs: The site publishes llms.txt and llms-full.txt for coding agents.

Pros

1. Free and permissive: MIT licence, no vendor account required.
2. Wide platform reach: Rare to find one framework spanning web, mobile, HarmonyOS and desktop.
3. Popular: Over 15,000 GitHub stars at review time.
4. Low model cost possible: The site shows one benchmark run of 60 tasks costing $0.59 in model fees; results vary by model.
5. Fits existing runners: Plays well with Playwright Test and Vitest.

Cons

1. Model setup required: You configure and pay for a vision model.
2. Non-deterministic steps: AI interpretation can vary run to run.
3. Docs span versions: v0.x and v1.x docs coexist; check which you are reading.
4. Self-managed: No vendor SLA or hosted grid.
5. Vision latency: Screenshot-based steps are slower than selectors.

Usage and Links (Example)

A YAML case adapted from the official site:

beforeEach:
  - gotoUrl: https://your-shop.example
cases:
  - name: "Find headphones under $100"
    steps:
      - aiAct: "Search for headphones and filter the results to prices under $100"
      - aiWaitFor: "The filtered results are shown"

Configure a model first (see the model selection guide), then run the case with the Midscene test runner.

Resources:

• Midscene.js website
• GitHub repository

Pricing

Midscene.js is free and open source under MIT. You pay the model provider you configure.

Recommended for?

1. Teams testing the same flow across web and mobile who want one natural-language format.
2. HarmonyOS app teams, a platform few tools support.
3. Engineers who want an open-source vision agent inside Playwright or Vitest.
4. Not a fit if you need a managed service with support contracts.

Conclusion

Midscene.js is one of the broadest open-source vision-driven testing frameworks, with YAML and JavaScript authoring and many platforms.

Pick a model carefully, keep critical assertions strict, and pin versions while the project evolves.