Agentic QA Goes Mainstream: AI Testing Agents in 2025

For the first half of 2025, “agentic testing” was something you assembled yourself: a model, an MCP server, a browser and some glue. Between mid-May and mid-July that changed. The largest testing platforms shipped named AI agents, the big coding assistants became autonomous enough to open pull requests on their own, and the browser-automation layer for agents raised serious money.

This essay covers that eight-week window. We skipped a June essay, so it’s a long one. The deeper theme is agentic QA going mainstream: what the vendors are actually shipping, how the pieces fit together, and how a QA team can adopt agents without giving up the determinism and accountability that make testing worth anything.

Coding agents that open pull requests

GitHub Copilot coding agent

On 19 May, at Microsoft Build, GitHub put the Copilot coding agent into public preview. This is the Project Padawan idea we mentioned in March. You assign a GitHub issue to Copilot, it works in a GitHub Actions-powered environment, and it opens a pull request for human review. It can run your tests and linters in that environment as part of its loop.

For QA, the important detail is the workflow, not the model. The agent’s output arrives as a normal pull request, subject to the same branch protection, required checks and code owners as any human change. Your test suite and CI gates are now the main quality control on an autonomous contributor.

Claude Code goes GA

On 22 May Anthropic launched Claude Opus 4 and Claude Sonnet 4, and Claude Code became generally available. It gained IDE integrations, GitHub Actions support for background tasks and an SDK for building custom agents. Combined with project instruction files and MCP servers, Claude Code is now a programmable test-automation engineer you can script.

Kiro and spec-driven development

On 14 July AWS launched Kiro in preview, an agentic IDE built around spec-driven development. A prompt becomes requirements with acceptance criteria, then a technical design, then sequenced implementation tasks, before code is written. For testers, that’s a striking product direction. A major vendor is betting that agents work better when they start from explicit acceptance criteria, which is exactly the argument for specifications we made in February.

OpenAI Codex and Gemini CLI

Two more launches completed the picture. On 16 May OpenAI introduced Codex, a cloud software-engineering agent that runs each task in an isolated sandbox preloaded with your repository, writes code, runs tests and proposes pull requests. On 25 June Google released Gemini CLI, an open-source terminal agent with MCP support and a generous free tier.

By early July, every major model provider offered an agent that can read a repository, run its tests and propose changes. Model quality still varies by task. The workflow, though, has converged on agent, sandbox, tests, PR and human review. That’s good news for QA: one set of practices for test rules, CI gates and review checklists applies whichever agent your developers prefer.

Testing platforms ship their own agents

BrowserStack AI

On 30 June BrowserStack launched BrowserStack AI, a suite of agents integrated across its products. They include a Test Case Generator Agent, which turns PRDs and user stories into test cases, and a Self-Healing Agent, which detects UI changes and repairs broken locators during execution on Automate and App Automate. BrowserStack also exposes agent capabilities through an MCP server, so they can be invoked from AI assistants.

mabl’s Test Creation Agent

In June mabl announced its Test Creation Agent. It takes natural-language test intent and autonomously builds end-to-end browser tests, complete with structured flows and reusable components, initially in early access.

Postman Agent Mode

At POST/CON 25 on 4 June, Postman announced Agent Mode, an AI assistant that works across collections, tests and documentation, along with tooling to generate MCP servers from APIs. For API testers, that means an agent that can build requests, write test scripts and debug failures inside the tool many teams already use. We cover it properly in Best API testing tools 2025.

The pattern across vendors

Vendor capability (mid-2025)	Agent role	Output	Human checkpoint
Test case generation (BrowserStack, mabl, others)	Analyst	Test cases / test outlines	Review coverage and expected results
Test creation (mabl, BrowserStack, testRigor, KaneAI)	Automation engineer	Executable tests	Review steps and assertions
Self-healing (BrowserStack, Testim, mabl, Functionize)	Maintainer	Locator repairs at runtime	Review heal logs
API agents (Postman Agent Mode)	API tester	Requests, scripts, docs	Review scripts and data
Coding agents (Copilot, Claude Code)	Developer	Code + tests in a PR	Code review and CI gates

Every row ends with a human checkpoint. The vendors that design those checkpoints well, with clear diffs, heal logs and reviewable outputs, will earn trust. The ones that hide them behind “fully autonomous” marketing will struggle once the first silent false pass reaches production.

The agent browser layer grows up

Browserbase and Stagehand

In June Browserbase announced a $40M Series B led by Notable Capital, along with Director, a no-code tool that turns natural-language instructions into Stagehand scripts. Stagehand is an open-source framework that blends deterministic Playwright code with AI primitives, so you choose per step whether to be precise or flexible:

import { Stagehand } from '@browserbasehq/stagehand';
import { z } from 'zod';

const stagehand = new Stagehand({ env: 'LOCAL' });
await stagehand.init();
const page = stagehand.page;

await page.goto('https://staging.example.test');                 // deterministic
await page.getByRole('link', { name: 'Sign in' }).click();       // deterministic
await page.act('dismiss the cookie banner if one appears');      // AI step
const cart = await page.extract({
  instruction: 'extract the cart subtotal and item count',
  schema: z.object({ subtotal: z.string(), items: z.number() }),
});                                                               // AI extraction, typed

That hybrid model is, in our view, the most promising design in agentic testing right now. Use code where the app is stable and the assertion must be exact. Use AI where the UI is unpredictable, such as cookie banners, A/B variants and third-party widgets. Use typed extraction with a schema to turn the agent’s perception into something you can assert on.

Browser Use

Browser Use, which raised its seed in March, kept growing as the open-source Python option for fully goal-driven agents. Between Playwright MCP for assistants, Stagehand for hybrid scripts and Browser Use for autonomous agents, the agent browser layer now covers the main use cases.

Meanwhile, in the deterministic layer

The frameworks underneath all these agents kept shipping too. Playwright 1.53 (June) added locator.describe(), which attaches a human-readable description to a locator that then appears in traces and reports. That small change makes both human and agent debugging easier:

const subscribe = page.getByTestId('btn-sub').describe('Subscribe button');
await subscribe.click(); // trace and report show "Subscribe button"

The same release added a configurable title for HTML reports. Playwright 1.54 (July) added a partitionKey cookie property for saving and restoring partitioned (CHIPS) cookies, a noSnippets option for the HTML reporter, and a --user-data-dir option so codegen sessions can reuse authentication state. It also removed Node.js 16 support and deprecated Node.js 18. See the release notes for details.

None of this makes headlines, but it matters. Agents that drive Playwright, whether through MCP, Stagehand or generated test files, inherit every improvement in locator semantics and trace quality. The better the deterministic layer explains itself, the better agents and humans can debug what went wrong. If you want agentic QA to work, keep your underlying framework current.

Securing the agent supply chain

On 24 June Snyk acquired Invariant Labs, the team behind the MCP tool-poisoning research we covered in April and the mcp-scan tool. A major application-security vendor buying an agent-security specialist signals that agent and MCP security is becoming a product category of its own. For QA teams running agents with access to staging environments, CI tokens and test accounts, scanning MCP configurations is now a reasonable baseline:

# scan locally configured MCP servers for suspicious tool descriptions
uvx mcp-scan@latest

What the surveys say

The TestRail Software Testing & Quality Report, featured in Software Testing Weekly in June, joined a growing set of surveys showing AI adoption rising in testing while trust and process maturity lag. That’s consistent with earlier data. PractiTest’s State of Testing 2025 found that close to half of respondents hadn’t adopted AI yet, and that privacy concerns were the leading barrier. The gap between vendor announcements and day-to-day practice is still wide. That isn’t a failure. It’s what early adoption looks like.

Running a two-week vendor bake-off

With several vendors now shipping agents, many teams will run a comparison this year. A bake-off that produces a defensible decision in two weeks looks like this.

Preparation (two days). Pick six to eight real test scenarios. Include two simple ones, two with dynamic content, one with a third-party widget or iframe, one requiring test data setup, and one deliberately ambiguous user story. Write the expected outcomes yourself, before any vendor sees them.

Generation (three days). Have each tool generate tests from the same inputs: the user stories for case generation, and the live staging app for test creation. Record time taken, the number of human corrections needed, and whether the generated assertions match your expected outcomes.

Stability (five days). Run every generated test at least twice a day against staging. Halfway through, ship a deliberate UI change, such as renaming a button and moving a form field, to exercise self-healing. Then ship a deliberate functional regression, a wrong total, to check that no tool heals its way past a real bug.

Scoring (two days). Score each tool on assertion accuracy, stability, heal correctness, the false pass on the injected regression, the reviewability of outputs, export options and projected cost at your volume.

The injected regression is the single most informative test of an AI testing tool. A tool that reports green on a wrong total has told you everything you need to know about its oracle.

Pricing models to understand before you sign

Agent features change the economics of testing tools in ways that are easy to miss during a trial:

Pricing model	How it scales	Watch for
Per seat	Number of users	Cheap to start; agents may reduce seats needed but vendors may re-price
Per test run / execution minute	Volume of runs	Agent-generated suites can grow fast and multiply cost
AI credits / tokens	Agent activity	Heavy exploration or repeated healing consumes credits quickly
Managed service	Coverage scope	Ownership of tests at contract end
Open source + your model bill	Your LLM usage	Costs move to your model provider invoice

Ask for a cost projection at three times your trial volume. If AI agents succeed, usage will grow, and that’s exactly when pricing surprises hurt most.

Skills the team will need

Agentic QA doesn’t remove the need for testers. It changes the skills mix. Over the next year the most valuable capabilities on a QA team are likely to be:

• Specification writing. Clear acceptance criteria and examples that both humans and agents can work from.
• Test design and oracle definition. Deciding what to check and what correct means.
• Prompt and instruction engineering for test agents. Writing the rules files, goals and constraints that steer agent behaviour.
• Reading code and CI configuration. Reviewing agent PRs and understanding pipeline gates.
• Security awareness for agents. Credentials, MCP permissions, prompt injection.
• Data literacy. Interpreting heal logs, flake statistics and mutation scores.

None of these is new to good testers. What’s new is that they’re no longer optional extras on top of writing scripts. They’re the core of the job.

A reference architecture for agentic QA

Pulling the pieces together, here’s how we’d structure agentic QA in a mid-sized product team in July 2025:

 Requirements ─► Spec / acceptance criteria (human-owned, reviewed)
                     │
                     ├─► Test case generation agent  ─► human review ─► test plan
                     │
 Code change ──► Coding agent PR ─► CI: unit + contract + E2E (deterministic) ─► human review ─► merge
                     │                         │
                     │                         └─► failures ─► triage agent (traces, logs) ─► suggested fix
                     │
 Nightly ─────► Exploratory agents (Browser Use / Stagehand / MCP) on staging ─► findings ─► human triage
                     │
 Runtime ─────► Self-healing on cloud grid ─► heal log ─► weekly human review ─► permanent locator fix

The principles behind it:

1. Humans own acceptance criteria. Agents can draft them. Humans approve them.
2. Deterministic tests gate merges. Agents can write them. CI runs them identically every time.
3. Agents explore, triage and maintain. These are the tasks where their speed helps and their non-determinism does no harm.
4. Every agent action leaves a reviewable trail. PRs, heal logs, session recordings, findings reports.

Measuring agentic QA

Teams adopting agents should track a few metrics from day one:

Metric	Why it matters
Agent PR acceptance rate	Are agent-authored changes good enough to merge?
Post-merge defect rate of agent PRs vs human PRs	Is quality holding up?
Heal events per week, and false heal rate	Is self-healing hiding regressions?
Exploratory findings per agent-hour	Are exploratory agents finding real issues?
Test suite mutation score over time	Are generated tests actually protective?
CI duration and cost	Agent-generated tests can bloat suites

Pros and cons of going agentic now

Pros

• Real productivity gains in test design, creation and maintenance.
• Vendor agents integrate with existing grids and platforms.
• Hybrid frameworks such as Stagehand make AI steps opt-in and typed.
• Coding agents work through PRs and CI, so existing controls still apply.

Cons

• Silent false passes from self-healing and AI assertions.
• Security exposure through MCP servers and agent credentials.
• Vendor lock-in for agent-generated test assets.
• Costs that scale with usage in ways that are hard to forecast.

What this means for your team

Choose one agentic use case per quarter. Test case generation from user stories is the safest first step. Self-healing with mandatory heal review is a reasonable second.

Make CI the contract with agents. If coding agents will open PRs in your repos, your required checks are your quality policy. Audit them now: what must pass, who must review, which paths need code owners.

Adopt hybrid scripts for awkward UIs. Where third-party widgets or A/B variants make tests flaky, try Stagehand-style AI steps for just those parts, with typed extraction for assertions.

Scan your MCP setup. Run an MCP scanner on developer and CI machines that use agents, pin server versions, and separate credentials by task.

Ask vendors about checkpoints. For every agent feature you evaluate, ask what the human checkpoint is, what the agent’s output looks like, and how you’d detect it being wrong.

Keep humans on the oracle. Agents can generate tests, run them and fix them. Deciding what correct means for your product is still the job.

Mid-2025 is the point where agentic QA stopped being a demo and became a purchasing decision. The teams that come out ahead will be those that adopt agents for leverage while keeping the final verdict reviewable, deterministic and owned by people.

Further reading on CheckOps

• Playwright MCP and MCP servers for agentic testing
• Playwright Test Agents and the truth about self-healing tests
• Best AI testing tools 2025