Playwright Test Agents: The Truth About Self-Healing Tests

For years “self-healing tests” was a commercial feature: a selling point for Testim, mabl, Functionize and others, and something open-source users approximated with plugins. In the four weeks to mid-October 2025, healing arrived in the most popular open-source web testing framework, as part of a full agentic loop. Playwright 1.56 shipped Playwright Test Agents, and with them a planner, a generator and a healer that any team can run with the coding agent it already uses.

That makes now a good moment for a clear-eyed look at self-healing tests: what they actually do, which kinds of healing are safe, and the policy that stops a healer from turning a real regression into a green build. We’ll also cover the rest of the window’s news: the Chrome DevTools MCP server, Gemini’s computer-use model, JUnit 6, and the npm worm that followed last month’s attacks.

Playwright Test Agents

Playwright 1.56, released on 6 October, introduced three agent definitions designed to guide an LLM through building Playwright tests. According to the Test Agents documentation:

• Planner explores the app and produces a Markdown test plan.
• Generator turns the Markdown plan into Playwright test files, verifying selectors and assertions live as it goes.
• Healer runs the suite and repairs failing tests.

You add the definitions to your project with one command, choosing your agent environment:

npx playwright init-agents --loop=vscode    # VS Code / Copilot
npx playwright init-agents --loop=claude    # Claude Code
npx playwright init-agents --loop=opencode  # opencode

The documentation notes that the definitions should be regenerated whenever Playwright is updated, because they’re collections of instructions and MCP tools that change with each release.

The artefacts are the point

What makes this design good for QA is that every stage produces a reviewable file in a predictable place:

repo/
  .github/                 # agent definitions
  specs/
    basic-operations.md    # human-readable test plan from the planner
  tests/
    seed.spec.ts           # seed test: fixtures, auth, environment setup
    create/add-valid-todo.spec.ts   # generated test, linked to its spec
  playwright.config.ts

The seed test is the key input. The planner runs it to perform all the initialisation your tests need, including global setup, project dependencies, fixtures and hooks, and uses it as the template for generated tests. Your conventions for auth, test data and fixtures therefore flow into everything the agents produce. Investing in a good seed test is the single most effective way to improve agent output.

The spec is a Markdown plan with scenarios, steps and expected results. It’s the natural human checkpoint. A tester reviews the plan, corrects the expected results and adds the edge cases the planner missed, all before any code exists.

The generated tests carry comments linking back to their spec and seed. That makes review straightforward: does the test do what the spec says, and does it assert what the spec expects?

What the healer does

When a test fails, the healer replays the failing steps, inspects the current UI to find equivalent elements or flows, suggests a patch such as a locator update, a wait adjustment or a data fix, and re-runs the test until it passes or guardrails stop the loop.

One detail in the documentation deserves attention. The healer’s output is either a passing test, or a skipped test if the healer believes the functionality is broken. That’s the right design instinct: a healer that can say “this looks like a real bug” rather than forcing green. It also means that a judgement normally made by a tester is now being made by a model, and that judgement needs review.

Other 1.56 additions

The same release added page.consoleMessages(), page.pageErrors() and page.requests() for retrieving recent console messages, errors and network requests. These are useful for both agents and humans diagnosing failures. It also added --test-list and --test-list-invert for running an explicit list of tests, and an HTML reporter option to disable the “Copy prompt” button introduced in 1.51.

The four kinds of self-healing

“Self-healing” covers very different mechanisms with very different risk profiles. It’s worth separating them.

Kind	How it works	Examples	Risk of hiding real bugs
Multi-attribute locator fallback	Records several attributes per element; if the primary fails, matches on the others	Healenium, Testim, mabl, Katalon smart locators	Low–medium
Semantic / AI locator matching	Uses a model to find the element that best matches the intent	testRigor-style plain-English steps, AI cloud-grid healing	Medium
Runtime flow adaptation	Agent changes the steps at run time to reach the goal	Goal-driven browser agents	High
Code repair	Agent edits the test source and proposes a change	Playwright healer, coding agents	Depends on review; high if auto-merged

Healenium is the best-known open-source example of the first kind for Selenium: it stores successful locators and element attributes, and when a locator fails it finds the most similar element and reports the healing. Commercial platforms combine the first two. The Playwright healer is the fourth kind, and its output is a code change. That makes it the most transparent kind when it goes through review, and the most dangerous when it doesn’t.

How healing hides bugs

Here are three realistic ways a healer can make a broken product look fine:

1. The wrong-button heal. A redesign renames “Place order” to “Review order” and moves the real submit button to a second step. The healer finds “Review order” as the closest match, clicks it, and the test’s remaining assertions happen to pass on the review page. Checkout is now untested, and nobody knows.
2. The weakened-assertion heal. A total shows £0.00 because of a pricing bug. The test fails on toHaveText('£54.00'). A code-repair healer “fixes” it by changing the assertion to toBeVisible(). Green build, broken pricing.
3. The timing heal. A page becomes slow because of a performance regression. The healer adds a longer timeout. The test passes, and a five-second regression ships.

None of these is hypothetical as a pattern. They’re the natural result of an optimiser told to “make the test pass”. The defence isn’t to avoid healing. It’s to constrain what a heal may change.

A self-healing policy that works

Here’s the policy we recommend for teams adopting the Playwright healer or any commercial healing feature.

Change type	Allowed automatically?	Requirement
Locator update, same role and accessible name	Yes, at run time	Logged; reviewed weekly
Locator update, different name or role	No	Human review before merge
Added or adjusted wait or timeout	No	Human review, plus a performance check
Assertion changed, weakened or removed	Never automatically	Human review with product context
Test data change	No	Human review
Test skipped as “functionality broken”	n/a	Immediate triage as a potential bug

Two rules carry most of the value. Assertions are sacred: no automated process may change what a test checks. Every heal is visible: in a PR diff, in a heal log, or both.

You can enforce the first rule mechanically. A CI check on healer pull requests can fail if any assertion line changed:

#!/usr/bin/env bash
# Fail if a PR from the healer modifies expect() assertions.
base=${BASE_REF:-origin/main}
if git diff "$base"...HEAD -- 'tests/**/*.spec.ts' \
   | grep -E '^[-+][^-+].*expect\(' >/dev/null; then
  echo "Healer PR changes assertions: human review with product context required" >&2
  exit 1
fi

That’s crude. It will also flag legitimate assertion changes, but that’s precisely when you want a human involved.

Running the agent loop well

Based on the design of Test Agents and what we’ve seen work with agentic test generation generally, here’s a practical workflow:

1. Write the seed test by hand. Make it use your real fixtures, auth setup and test-data helpers. Keep it small and clean, because agents imitate it.
2. Ask the planner for one flow at a time. “Generate a plan for guest checkout with a discount code” works better than “plan tests for the whole app”.
3. Review the spec like a test design. Fix expected results, add negative and boundary cases, remove trivial checks. This is where tester expertise has the most leverage.
4. Generate, then review like code. Check locators against your conventions, prefer role-based locators, and make sure assertions match the spec.
5. Run the healer on failures in a branch, never on main. Treat its output as a proposal.
6. Triage every “skipped as broken” immediately. Either it’s a real bug, which is a great catch, or the healer is wrong, which is useful calibration.
7. Regenerate agent definitions on every Playwright upgrade, as the docs advise.

Also in the window

Chrome DevTools MCP

On 23 September the Chrome team launched a public preview of the Chrome DevTools MCP server, which lets AI coding agents control and inspect a live Chrome instance: reading console and network activity, and recording and analysing performance traces. Playwright MCP gives agents a testing view of the page. Chrome DevTools MCP gives them a debugging and performance view. An agent can now be asked to “load the product page, record a trace and tell me what’s hurting LCP”, which brings performance investigation into the same agent workflow as functional testing. It complements the browser-performance tools in our performance testing tools roundup.

Gemini 2.5 Computer Use

On 7 October Google DeepMind released the Gemini 2.5 Computer Use model in preview through the Gemini API. It’s optimised for browser control, with promising mobile results. The model card reports 65.7% on Online-Mind2Web and 79.9% on WebVoyager in Browserbase-run evaluations, and 69.7% on AndroidWorld. A year after we covered Claude’s first computer-use release at under 15% on OSWorld in November 2024, browser-control models are clearly far better. Note that these are different benchmarks, so the figures aren’t directly comparable. They’re still far from deterministic, which is why the hybrid architecture in our February essay still holds.

JUnit 6

JUnit 6.0.0 was released on 30 September. It requires Java 17, unifies the version numbers of Platform, Jupiter and Vintage, adopts JSpecify nullability annotations and supports Kotlin suspend test methods. For teams on JUnit 5 with a modern JDK the upgrade is mostly mechanical. Check the upgrade notes for removed deprecated APIs.

Shai-Hulud: the npm worm

Days after our supply-chain essay, a self-replicating npm worm named Shai-Hulud spread through the registry. On 23 September CISA issued an alert saying more than 500 packages had been compromised. The worm harvested npm tokens, GitHub tokens and cloud credentials, then used stolen npm access to publish infected versions of other packages automatically. CISA’s advice matches the playbook from last month: review lockfiles, pin to versions released before 16 September, rotate credentials, and enforce phishing-resistant MFA. If you hadn’t yet scoped secrets in your test pipelines, this was the reminder.

Healing in practice: a worked example

Here’s how the policy plays out on a realistic change. Suppose a team ships a redesign of its account settings page, and four E2E tests fail overnight.

Failure 1. The “Save changes” button is now labelled “Save”. The healer updates getByRole('button', { name: 'Save changes' }) to getByRole('button', { name: 'Save' }). The role is the same and the name is a close match. Under the policy, this is a run-time-allowed heal: it’s logged, and someone confirms it in the weekly review.

Failure 2. The email field moved into a new “Contact” tab. The healer adds a step to click the tab before filling the field. That’s a flow change, so it needs human review. A tester confirms it matches the new design and approves.

Failure 3. A success toast no longer appears after saving. The healer’s proposed patch replaces await expect(page.getByText('Settings saved')).toBeVisible() with a check that the button is enabled. That’s an assertion change, so the CI check blocks it. On investigation, the toast was removed by mistake in the redesign and users get no confirmation. That’s a real UX bug, caught because the policy refused to let the healer weaken the assertion.

Failure 4. The healer marks a test as skipped because the “Delete account” flow returns an error. Triage confirms a backend regression. Another real bug, caught because the healer was allowed to say “broken” rather than forced to find a path to green.

Two of the four failures were real defects. A permissive healer would have hidden both. A constrained one found them and still saved the team the tedium of updating locators by hand.

How Test Agents compare to commercial healing

	Playwright Test Agents	Commercial AI testing platforms
Cost	Open source; you pay for your model and agent	Subscription, often usage-based
Output	Plain Playwright code and Markdown specs in your repo	Platform-managed tests, export varies
Healing style	Code repair, proposed as changes	Mostly run-time locator healing with logs
Review model	Your normal PR review	Platform dashboards and heal reports
Lock-in	Low	Medium to high
Setup effort	Medium: seed test, agent environment, policy	Low to medium
Best for	Teams already on Playwright with code review culture	Teams wanting managed infrastructure and low-code authoring

Neither is universally better. The open-source route gives you transparency and ownership at the cost of setup. Commercial platforms give you convenience, grids, dashboards and support at the cost of dependency. Both need the same healing policy.

Pros and cons of Playwright Test Agents

Pros

• Open source and agent-agnostic: VS Code, Claude Code and opencode loops.
• Every stage produces a reviewable artefact: specs, tests, diffs.
• Seed tests encode your conventions into generated tests.
• The healer can flag functionality as broken rather than forcing a pass.

Cons

• Output quality depends on your model, seed test and app accessibility.
• Model and agent costs scale with usage.
• Agent definitions must be kept in sync with Playwright versions.
• Without policy, healed tests can drift away from what they were meant to check.

What this means for your team

Try Test Agents on one flow this month. Write a strong seed test, plan a single flow, review the spec carefully, generate, review and run. Measure the review effort honestly.

Adopt a written healing policy before enabling any healer. Use the table above as a starting point, and make “assertions are sacred” non-negotiable.

Instrument heals. Track heal events, heal types and false-heal discoveries. A rising heal rate is a signal about UI churn or test design, not just a maintenance saving.

Treat “skipped as broken” as a bug report. Route it to triage with the same priority as a failing test.

Use the right MCP for the job. Playwright MCP for functional exploration, Chrome DevTools MCP for debugging and performance analysis. Both should run against staging, with scoped credentials.

Keep accessibility high. Agents and healers work best on apps with good roles and accessible names. Accessibility work now pays off twice.

Self-healing tests were always a trade: less maintenance in exchange for less certainty. Playwright’s design, with plans in Markdown, tests in your repo, heals as diffs and the option to declare a feature broken, makes that trade explicit and reviewable. The teams that benefit will be the ones that keep it that way. For a wider comparison of agentic tools, see Best AI testing tools 2025.

Further reading on CheckOps

• Playwright MCP and MCP servers for agentic testing
• Agentic QA goes mainstream
• Playwright vs Cypress vs Selenium: the BiDi shift