Mutation Testing for AI-Written Tests: Radar Vol 34

AI coding agents are very good at producing tests that pass. That is exactly the problem. A test that passes no matter what the code does gives you a green build and no information.

On 15 April 2026, Thoughtworks published Technology Radar Volume 34, and one of its quieter entries is one of the most useful for testers. Mutation testing sits in the Trial ring, with a justification that reads like a description of 2026’s testing problem: “With AI-generated test cases now commonplace, mutation testing acts as a reinforcement layer for catching ‘perpetually green’ tests — those that pass regardless of logic changes due to missing assertions or decoupled mocks.”

This essay explains what mutation testing is, why it suddenly matters again, and how to add it to a codebase where a large share of the tests were written by an AI. It builds on our look at Meta’s Just-in-Time tests, which use mutants to guide test generation, and our essay on AI coding agents writing tests.

What is mutation testing?

Mutation testing checks the quality of your tests by deliberately breaking your code and seeing whether the tests notice.

1. A tool makes a small change to the code, called a mutant. For example, it changes > to >=, replaces && with ||, removes a function call, or makes a function return a constant.
2. It runs your tests against the mutated code.
3. If at least one test fails, the mutant is killed. Good.
4. If all tests still pass, the mutant survived. Your tests did not detect a real change in behaviour.

The mutation score is the share of mutants killed. Unlike line coverage, which only tells you code was executed, the mutation score tells you whether the tests would have caught a change in that code.

Thoughtworks puts it this way: mutation testing shifts “the focus from how much code is executed to how much code is actually verified”. The aim is that “a passing test suite is a reliable signal of functional correctness, rather than simply a report of which lines were executed.”

Why Radar Vol 34 brought mutation testing back

Mutation testing is not new. What is new is the volume of tests written by machines.

Theme: retaining principles, relinquishing patterns

One of Volume 34’s themes is “Retaining principles, relinquishing patterns”. Thoughtworks writes that while assembling the edition, “we found ourselves returning to many established techniques, from pair programming to zero trust architecture, and from mutation testing to DORA metrics.” It calls this “a necessary counterweight to the speed at which AI tools can generate complexity.”

Theme: putting coding agents on a leash

Another theme describes feedback sensors for coding agents: “deterministic quality gates — compilers, linters, type checkers and test suites — integrated directly into agent workflows so failures trigger auto-correction before human review.” The Radar lists cargo-mutants “and other mutation testing tools” as examples.

That is the key idea for 2026. Mutation testing is not just a report you read once a quarter. It is a signal you can feed back to the agent that wrote the tests, so it can improve them before a human looks.

How AI-written tests go wrong

These are the patterns that mutation testing commonly exposes in machine-written suites. Each one produces a test that passes and proves little.

1. Missing or weak assertions

test('calculates invoice total', () => {
  const total = calculateTotal(items, { taxRate: 0.2 });
  expect(total).toBeDefined();
});

Any change to calculateTotal that still returns a number passes. A mutant that drops the tax calculation survives.

2. Asserting on the mock, not the behaviour

test('sends welcome email', async () => {
  const mailer = { send: vi.fn() };
  await registerUser({ email: 'a@example.com' }, mailer);
  expect(mailer.send).toHaveBeenCalled();
});

If registerUser sends the wrong template, to the wrong address, the test still passes. Thoughtworks calls this “decoupled mocks”.

3. Snapshot everything

Large snapshots generated in one pass and accepted without review tend to get updated whenever they fail. Mutants that change output may be “killed”, but the next agent run simply updates the snapshot.

4. Testing the implementation back to itself

Agents sometimes compute the expected value with the same logic as the code under test. The test and the code agree because they are the same code.

5. Happy path only

Boundary conditions are where mutants like > to >= live. If the agent only tested typical values, those mutants survive.

Mutation testing tools in 2026

The Radar names three tools. All are open source and mature.

Tool	Ecosystem	Notes
Stryker	JavaScript/TypeScript (StrykerJS), C# (Stryker.NET), Scala (Stryker4s)	Works with Jest, Vitest, Mocha and others; HTML report
Pitest (PIT)	Java and the JVM	Maven and Gradle plugins; fast bytecode mutation
cargo-mutants	Rust	Runs cargo test against each mutant

Other ecosystems have options too, such as mutmut for Python, but the Radar entry focuses on these three.

How to add mutation testing to a project

Here is a practical rollout for a codebase with a lot of AI-written tests.

Step 1: start with core domain logic

Do not mutate the whole codebase on day one. The Radar specifically says “particularly in core domain logic”. Pick the modules where a silent bug is expensive: pricing, permissions, billing, data transformations.

Step 2: run it locally first

For a TypeScript project using Vitest or Jest, StrykerJS has an interactive initialiser:

npm init stryker@latest
npx stryker run

Point the mutate setting at your domain folder, for example src/domain/**/*.ts. For Java with Maven and JUnit 5, the PIT Maven plugin runs with:

mvn org.pitest:pitest-maven:mutationCoverage

For Rust:

cargo install cargo-mutants
cargo mutants

Step 3: read the survivors, not the score

The score is a summary. The value is in the list of surviving mutants. For each survivor, ask: should a test have caught this? Most of the time the answer is yes, and the fix is a better assertion. Sometimes the mutant is “equivalent”, meaning the change does not alter behaviour, and you can ignore it.

Step 4: feed survivors back to the agent

This is the 2026 workflow. Give your coding agent the surviving mutants and a clear instruction:

These mutants survived in src/domain/pricing.ts. For each, add or strengthen a test that fails on the mutant and passes on the original. Do not change production code. Do not weaken existing assertions.

Then run mutation testing again. This loop is exactly what Thoughtworks means by a feedback sensor: a deterministic tool tells the agent what its tests missed, and the agent corrects itself before review.

Step 5: add it to CI carefully

Mutation testing is slow because it runs tests many times. Common ways to keep it practical:

• Run it only on changed files in pull requests, and on the full domain folder nightly.
• Use incremental modes where the tool supports them, so unchanged code is not re-mutated.
• Set a threshold for critical modules, and treat a drop as a warning before you make it a hard gate.

Step 6: review what the agent changed

Agents under pressure to kill mutants can write over-specified tests that break on every refactor. Review new tests for intent: do they check behaviour the business cares about, or just implementation details?

A worked example: from survivor to strong test

Here is what the loop looks like on a small piece of code. Suppose an agent wrote this function and test:

// src/domain/shipping.ts
export function shippingFee(subtotal: number, isMember: boolean): number {
  if (isMember || subtotal >= 100) return 0;
  return 7.5;
}

// src/domain/shipping.test.ts
test('members get free shipping', () => {
  expect(shippingFee(20, true)).toBe(0);
});

test('non-members pay shipping on small orders', () => {
  expect(shippingFee(20, false)).toBe(7.5);
});

Coverage for shippingFee is 100%. Every line runs. Now run mutation testing. A typical tool will try mutants such as:

• subtotal >= 100 becomes subtotal > 100
• subtotal >= 100 becomes true or false
• isMember || ... becomes isMember && ...
• return 7.5 becomes return 0

The >= to > mutant survives, because no test checks a subtotal of exactly 100. The “condition becomes false” mutant for the subtotal check also survives, because no test checks a large order from a non-member. Both are real gaps. The fix is two small tests:

test('non-members get free shipping at exactly 100', () => {
  expect(shippingFee(100, false)).toBe(0);
});

test('non-members pay shipping just under the threshold', () => {
  expect(shippingFee(99.99, false)).toBe(7.5);
});

Rerun, and those mutants are killed. Notice that this is a business question as much as a technical one. Is free shipping at “100 or more” or “more than 100”? A surviving boundary mutant is often a prompt to check the requirement, not just the test.

A starter StrykerJS configuration

For a TypeScript project using Vitest, a minimal stryker.config.json might look like this:

{
  "testRunner": "vitest",
  "mutate": ["src/domain/**/*.ts", "!src/domain/**/*.test.ts"],
  "reporters": ["html", "clear-text", "progress"],
  "thresholds": { "high": 80, "low": 60, "break": 50 },
  "incremental": true
}

The break threshold fails the run when the score drops below that number, which is how you turn the report into a gate. The threshold values here are placeholders; pick numbers based on your own baseline after the first run, not on anyone’s recommended target. incremental reuses previous results so unchanged code is not mutated again, which makes pull request runs much faster.

Mutation score vs code coverage

	Line coverage	Mutation score
Measures	Which lines ran	Whether tests detect changes
Cost	Cheap	Expensive, many test runs
Gameable by AI?	Easily, call code without asserting	Much harder, needs real assertions
Best use	Spot untested areas	Judge whether tests verify behaviour

Coverage is still useful as a cheap map of what is untested. But in a codebase where an agent can raise coverage to a high number in minutes, coverage alone is a weak quality gate. Mutation score is much harder to game.

Common objections

“It’s too slow”

It is slower than unit tests. Scope it to core logic, run it incrementally, and schedule full runs overnight. The time you spend is small compared with debugging a production bug a passing suite missed.

“Equivalent mutants waste time”

Some mutants do not change behaviour. Mark them and move on. Modern tools generate fewer of these than older ones, and reviewing a handful is a good exercise in understanding the code.

“Our coverage is already 90%”

Coverage says the code ran. It does not say the tests would notice if it were wrong. If AI wrote most of those tests, mutation testing is the quickest way to find out what that 90% is worth.

Pros and cons

Pros

• Exposes tests that pass regardless of behaviour, including AI-written ones
• Gives a concrete, actionable list of gaps
• Works as an automated feedback loop for coding agents
• Mature open-source tools for major ecosystems

Cons

• Slow on large codebases without scoping
• Equivalent mutants need human judgement
• Can push agents towards over-specified tests if unchecked

Where mutation testing fits in a 2026 test strategy

Thoughtworks also placed browser-based component testing in Trial in the same volume, noting that running component tests in a real browser with tools such as Playwright is now “a viable and often preferable approach”. Together these two entries describe a sensible direction: test behaviour in realistic environments, and verify that tests actually detect changes.

Mutation testing belongs at the unit and component layers, on core logic. It does not replace end-to-end checks, exploratory testing or production observability, which we covered in shift left and shift right. It does make sure the cheapest, fastest layer of your pyramid is telling you the truth.

Metrics worth tracking

Once mutation testing runs regularly, a few simple numbers tell you whether it is working:

• Mutation score per critical module. Track the trend, not just the value. A sudden drop after a large agent-authored change is a signal to review its tests.
• Surviving mutants per pull request. If new code regularly arrives with survivors, improve your prompts or agent instructions so tests are written with assertions on outcomes and boundaries.
• Time to run. If the job gets slow enough that people skip it, narrow the scope or move more of it to nightly runs.
• Escaped defects in mutated modules. The real test of the practice is whether fewer bugs reach production from the code you protect.

Keep the dashboard small. The point is a trustworthy green build, not a new vanity metric.

What this means for your team

If AI writes a meaningful share of your tests, add mutation testing to your core domain modules this quarter. Start locally, read the surviving mutants, and feed them back to the agent with clear instructions. Then make it a scheduled check, and consider a pull request gate for the modules that matter most.

The broader point from Radar Vol 34 is that AI makes old disciplines more valuable, not less. A green build should mean something. Mutation testing is one of the few tools that can tell you whether it does.

Start small, measure the score on the code that matters most, and let the surviving mutants tell you where to look next.

Frequently asked questions

What is a “perpetually green” test?

Thoughtworks uses the term for tests that pass regardless of logic changes, usually because of missing assertions or mocks that are not connected to real behaviour. Mutation testing finds them by changing the code and checking whether any test fails.

What ring is mutation testing in on Technology Radar Vol 34?

Trial, published on 15 April 2026. Trial means “worth pursuing”, to be tried on a project that can handle the risk.

Which mutation testing tool should I use?

Stryker for JavaScript, TypeScript, C# and Scala; Pitest for Java and the JVM; cargo-mutants for Rust. These are the three tools Thoughtworks names in the Radar entry.

Is mutation score better than code coverage?

They measure different things. Coverage shows what code ran; mutation score shows whether tests would detect a change. For AI-written tests, mutation score is a much stronger signal, but coverage remains a cheap way to find untested areas.

Can AI agents fix surviving mutants?

Yes, and this is a good use of them. Give the agent the list of surviving mutants and ask for tests that kill them without changing production code, then review the new tests and rerun the tool.

Further reading on CheckOps

• Meta JiTTests: just-in-time tests for agentic code
• AI coding agents writing tests: Claude Code and Copilot
• LLM testing, evals and vibe coding quality
• Shift left, shift right: testing and observability
• Testing vs checking: why your software needs both