AI Coding Agents Writing Tests: Claude Code and Copilot

Until early 2025, “AI writes the tests” meant autocomplete. You typed it('should and an assistant suggested the rest. In the past few weeks that changed. AI coding tools became agents: they read the codebase, run the test suite, see the failures, edit code and run the tests again, in a loop, with less and less human input between steps.

This essay covers mid-February to mid-March 2025, when Claude Code arrived, Copilot’s agent mode rolled out, and Playwright started putting “copy prompt” buttons in its reports. The deeper theme is LLM test automation. What happens to test quality when the thing writing the tests is also the thing writing the code, and how do QA teams keep that loop honest?

The month agents learned to run tests

Claude Code

On 24 February Anthropic released Claude 3.7 Sonnet alongside Claude Code, a command-line agent offered as a limited research preview. Claude Code lives in the terminal. It can search and read files, edit code, run commands, including your test runner, and commit to git, asking permission for actions according to configurable rules. Anthropic’s announcement specifically described it as able to write and run tests.

A typical session looks like this:

$ claude
> The discount calculation fails for orders over 1,000 when a coupon is applied.
  Reproduce it with a failing test first, then fix it. Run the full unit suite before finishing.

● Read src/pricing/discount.ts
● Read src/pricing/discount.test.ts
● Edit src/pricing/discount.test.ts   (+14 lines: "applies coupon after tier discount above 1000")
● Bash  npm test -- discount         → 1 failed
● Edit src/pricing/discount.ts        (order of operations fix)
● Bash  npm test                      → 212 passed

That workflow, red test first, then fix, then full suite, is exactly what a disciplined engineer would do. The agent does it in a minute. The question for QA is what happens when the instruction isn’t that disciplined.

Copilot agent mode

GitHub announced Copilot agent mode in VS Code Insiders on 6 February and brought it to more users later in the month. It can iterate on its own output, run terminal commands and fix errors it detects, test failures included. GitHub also previewed Project Padawan, an autonomous agent that would take an assigned issue and return a tested pull request.

Cursor and the IDE agents

Cursor’s agent features, used by Karpathy in his vibe coding post, work in the same loop inside the editor: propose edits across files, run commands, read output, continue. Windsurf, Cline and others offer similar workflows. By March the pattern is universal: an LLM in a loop with your tests as its feedback signal.

Playwright’s “Copy prompt”

The test-tooling side noticed. Playwright 1.51, released on 6 March, added a Copy prompt button to the HTML report, trace viewer and UI mode. When a test fails, one click copies a ready-made prompt with the error, the relevant test code and context, for pasting into an AI assistant. It’s a small feature with a big signal. Test frameworks are now designing their failure output to be consumed by LLMs as well as by humans.

The same release added IndexedDB support to storageState, which matters for apps that keep auth tokens there:

await page.context().storageState({ path: 'auth.json', indexedDB: true });

The core problem: who tests the tester?

When a human writes code and a different human writes tests, the tests encode an independent understanding of what the code should do. When the same agent writes both, in the same session, from the same context, that independence disappears. Several failure modes follow.

1. Tests that mirror the implementation

Ask an agent to “add tests for applyDiscount” and it will read applyDiscount and write tests that pass against it. If the function has a bug, the tests faithfully encode the bug. Coverage goes up and confidence goes up, but correctness doesn’t move.

2. Making the tests pass by changing the tests

An agent told to “make the tests pass” has two options: fix the code or weaken the tests. Without explicit instructions it sometimes picks the second: loosening an assertion, adding a skip, or changing the expected value to match the actual one. In a large diff that’s easy to miss.

- expect(total).toBe(1080.00);
+ expect(total).toBeCloseTo(1100, -2);

That diff looks like a precision tweak. It’s actually accepting a 20-unit error.

3. Over-mocking

Agents reach for mocks to isolate units, which is often right, but they can mock away the very behaviour under test. A test of a payment retry policy that mocks the retry helper tests nothing.

4. Volume without value

Agents can produce hundreds of tests quickly. Many are trivial: getters, constructors, framework behaviour. They slow the suite, add maintenance and inflate coverage numbers that leadership may then rely on.

5. Flaky end-to-end tests

For UI tests, agents often generate brittle selectors, fixed waits and order-dependent steps unless steered toward good practice. The generated test passes once, in the agent’s session, and flakes in CI.

Guardrails that work

None of these problems is new. They’re the same problems as with junior engineers, accelerated. The guardrails are familiar too, just made explicit and, where possible, automated.

Write the rules down where the agent reads them

Agents read project instruction files: CLAUDE.md for Claude Code, .github/copilot-instructions.md for Copilot, and rules files for Cursor. Put your testing standards there:

# Testing rules (read before changing tests)
- Never weaken, skip or delete an existing assertion to make a test pass.
  If you believe an assertion is wrong, stop and explain why.
- For bug fixes: write a failing test that reproduces the bug BEFORE changing code.
- Derive expected values from the spec or ticket, not from running the code.
- Do not mock the module under test. Mock only network, clock and randomness.
- E2E: use getByRole / getByLabel locators. No waitForTimeout. No CSS nth-child.
- Run `npm test` and `npm run lint` before declaring done.

These files are cheap and surprisingly effective. They’re also a good forcing function for teams that never wrote their testing standards down.

Protect test files in review

Make changes to existing assertions visible. Some teams add a CI check that flags pull requests modifying assertions in existing tests, so a human must explicitly approve them:

# flag PRs that change existing expect(...) lines in test files
git diff origin/main...HEAD -U0 -- '**/*.test.*' '**/*.spec.*' \
  | grep -E '^-\s*.*expect\(' && echo "::warning::Existing assertions modified - needs QA review"

It’s crude, but it catches the “make it pass by changing the test” pattern.

Separate the author of tests from the author of code

The strongest guardrail restores independence. Some options, from lightest to heaviest:

• Have the agent write tests from the ticket or spec only, in a fresh session without the implementation in context, then implement the code in a second session.
• Have a human write the acceptance tests, or at least the expected values, and let the agent write the implementation and unit tests.
• Use mutation testing to check whether agent-written tests actually detect faults. A suite that kills few mutants is decorative.

# JavaScript/TypeScript: Stryker
npx stryker run
# Python: mutmut
mutmut run && mutmut results

Mutation score is one of the best available metrics for AI-generated test quality, because it measures exactly what we care about: would these tests catch a bug?

Keep batches small

DORA’s 2024 finding, which we discussed in November, applies directly. Agents make large diffs effortless, and large diffs are where weakened assertions hide. Ask agents to work in small, reviewable steps, and reject mega-PRs regardless of who or what wrote them.

A worked example: the two-session workflow

Here’s how restoring independence looks in practice for a small feature, “free shipping for orders over 50 in the EU, except for oversized items”.

Session 1: tests from the spec only. Start a fresh agent session with no access to the implementation branch. Give it the ticket and the test conventions:

Here is ticket SHIP-142 (pasted). Write Vitest tests in src/shipping/freeShipping.test.ts
for a function qualifiesForFreeShipping(order): boolean that does not exist yet.
Cover: threshold boundary (49.99 / 50.00 / 50.01), EU vs non-EU, oversized items,
mixed carts, currency other than EUR. List the cases first. Do not implement the function.

A human reviews the list of cases before any code is written. That’s the cheapest moment to catch a misunderstanding, such as “is the threshold before or after discounts?”. The answer goes into the ticket, not just the test.

Session 2: implementation against the tests. A second session, which can be the same agent, implements the function with the instruction that tests may not be modified. If the agent believes a test is wrong, it stops and says so instead of editing it.

Gate: mutation check. Run mutation testing on the new module. If mutants survive, for example if flipping >= to > isn’t caught, the case list was incomplete. Add a test, and add a line to the ticket template so the next feature covers boundaries explicitly.

The whole cycle takes little longer than letting one agent do everything in one go. The difference is that the tests now represent the ticket, not the code.

Reviewing an agent-written test: a quick checklist

When an agent-generated test lands in a pull request, these questions catch most problems in a minute or two:

Question	Red flag
Where did the expected values come from?	Values that look copied from a console.log of actual output
Would this test fail if the feature were broken?	Assertions only on non-null, length or “toBeDefined”
What’s mocked?	The module under test, or its key collaborator, is mocked
Is anything skipped or loosened?	.skip, .only, toBeCloseTo with coarse precision, broadened regexes
Is the test deterministic?	Real clock, random data without a seed, network calls, fixed sleeps
Is it worth keeping?	Tests of getters, framework behaviour or constants

Keeping a checklist like this next to the review queue makes the review habit concrete. The goal isn’t to distrust agents. It’s to review their tests with the same scepticism we’d apply to a new colleague’s first week of work.

Where agent-written tests shine

It would be wrong to come away negative. Used well, coding agents are the best thing to happen to test coverage in years:

• Characterisation tests for legacy code. Before refactoring an untested module, ask the agent to write tests that capture current behaviour, explicitly as a snapshot of what the code does, not what it should do. That makes refactoring safe.
• Bug reproduction. “Write a failing test that reproduces this stack trace” is a task agents do very well, and it’s the most valuable test you can write.
• Boundary and edge-case enumeration. Agents are good at listing edge cases humans forget: empty strings, Unicode, time zones, leap years, integer overflow.
• Test data builders and fixtures. Tedious scaffolding that slows humans down.
• Migrating test frameworks. Converting Jest to Vitest, or Cypress to Playwright, file by file with the suite run after each change.
• Explaining failures. Feeding a trace or the Playwright “Copy prompt” output to an assistant speeds up triage.

A comparison of the main agentic coding tools for testing

Tool (Mar 2025)	Where it runs	Test-loop strengths	Things to watch
Claude Code	Terminal	Runs any command; good at multi-file test + fix loops; CLAUDE.md rules	Research preview; permission settings matter
GitHub Copilot agent mode	VS Code	Integrated with editor and terminal; repo instructions file	Preview; behaviour varies by selected model
Cursor agent	Cursor IDE	Fast multi-file edits; rules files	Easy to accept large diffs without reading them
Qodo	IDE + PR	Focused on test generation and code review	Generated tests still need assertion review

Capabilities change monthly, so treat this table as a snapshot. What’s durable is the pattern: an LLM, a set of tools including your test runner, and a loop.

Testing frameworks are adapting

Beyond Playwright’s prompt button, the frameworks themselves are getting friendlier to AI-driven workflows:

• Cypress 14.1 (late February) made WebDriver BiDi the default protocol for Firefox 135+, aligning with the standards shift we described in September. Standard protocols make it easier for every tool, AI-driven ones included, to automate browsers consistently.
• Accessibility-tree representations, such as Playwright’s aria snapshots from November, give LLMs a compact, semantic view of a page. We expect them to become the default way agents “see” web apps. More on that in April.

The QA role in an agent-written codebase

If agents write more of the code and more of the tests, QA work shifts toward:

1. Owning the oracle. Deciding what correct means: acceptance criteria, expected values, invariants. That’s the part agents can’t infer from code.
2. Auditing test quality. Mutation scores, assertion reviews and flake analysis. Treat the test suite as a product with quality metrics of its own.
3. Designing agent guardrails. Writing and maintaining the instruction files, CI checks and review policies that keep agents honest.
4. Exploratory and risk-based testing. Finding the problems nobody thought to specify, which is what neither humans nor agents can derive from existing code.

That’s a more senior, more leveraged role than writing test scripts by hand, and it’s where we expect demand to grow. We take stock of how that played out in our 2025 year-in-review.

Pros and cons of agent-written tests

Pros

• Massive speed-up on scaffolding, fixtures and boilerplate.
• Higher baseline coverage on previously untested code.
• Faster bug reproduction and triage.
• Lowers the barrier for developers who avoided writing tests.

Cons

• Loss of independence between code and tests.
• Risk of weakened assertions and mirror tests.
• Test-suite bloat and slower pipelines.
• False confidence from inflated coverage numbers.

What this means for your team

Write a testing rules file this week. Whatever agent your developers use, put your testing standards where it will read them. Start with “never weaken assertions to make tests pass”.

Require red-first for bug fixes. Make “failing test first” a review requirement for every bug fix, human or agent.

Add mutation testing to one critical module. Run Stryker, PIT or mutmut nightly on your pricing, auth or permissions code. Watch the score when agent-written tests arrive.

Flag assertion changes. Add a lightweight CI warning when existing assertions change, and route those PRs to someone who understands the domain.

Measure the right things. Track mutation score, escaped defects and flake rate rather than test count or line coverage, which agents can inflate effortlessly.

Use the agents. None of this argues for avoiding coding agents. They’re excellent at the tedious parts of testing. Use them deliberately, with humans owning the definition of correct.

The test suite has always been a team’s executable understanding of what the software should do. In the agent era, it may become the only such understanding anyone reviews. That makes test quality the single most important thing QA teams can protect.

Further reading on CheckOps

• Browser agents vs test automation: Operator and BDD
• Playwright MCP and MCP servers for agentic testing
• LLM testing and evals after the vibe-coding summer