Test Impact Analysis: How Agentic Coding Breaks CI

For years, the bottleneck in software delivery was writing code. In 2026, at companies that have gone all in on coding agents, that has flipped. Code is cheap, pull requests multiply, and the pressure moves downstream to review, CI and verification.

Two engineering posts from September 2026 show what that looks like in practice. Anthropic described how its test impact analysis service nearly buckled under a 25x rise in CI jobs. Spotify described how the volume of change “increased faster than some of our verification controls could adapt.” Both were featured together in Software Testing Weekly #329 under the heading “Quality at AI Speed.”

This essay explains test impact analysis, walks through what Anthropic built and why, and turns both posts into practical advice for teams whose CI is starting to feel the strain.

What is test impact analysis?

Test impact analysis (TIA), also called test selection, decides which tests to run for a given change instead of running the whole suite on every pull request. It uses signals such as which files or packages changed, which tests cover them and how those tests behaved in the past.

The alternative, running everything on every change, is simple and safe until it isn’t. As Anthropic’s Sachin Malhotra puts it: “This works up to a point, but doesn’t scale: CI gates get increasingly long, expensive, and untrustworthy.”

Why agents make TIA more important

The Anthropic post adds a reason specific to agents. “Humans are great at determining which test failures don’t apply to them while agents will require more context and direction. When they get a specific set of valid tests, they can self-verify and iterate more effectively.”

In other words, a noisy CI run with unrelated failures costs a human a few minutes of judgement. It can send an agent down the wrong path entirely.

Anthropic’s numbers

The post, Agentic coding is straining CI, published on 14 September 2026, gives the context:

• Anthropic engineers “on average ship 8x as much code per quarter as they did from 2021-2025,” with Claude authoring 80% of that code, according to figures Anthropic links in the post.
• The number of tests “grew 10x” while headcount grew only nominally.
• CI jobs increased 25x over six months. Not every test runs on every PR, which is the point of TIA.

Malhotra’s summary of the lesson: “always plan for the exponential.”

How Anthropic’s test selection works

Anthropic’s service is deterministic and has two parts:

• A listener that records test results from every CI run.
• A selector that reads the test result history and decides which tests run on each pull request, “based on past performance and package relevance.”

The two must stay in sync. When CI jobs arrive every second, the listener falls behind. The post gives a vivid example: “20 minutes of listener lag can translate into tens of thousands of test updates not being applied to the selector.” The consequences:

• A bad change merges, and a test fails for everyone, triggering unnecessary investigations.
• A dependency starts flaking, and flaky reds block merges.
• A test gets fixed or added, but won’t run until the listener catches up, risking a regression.

The original design ran as a single process, because keeping a per-test history required a single writer. That prevented horizontal scaling.

Three patches and a redesign

The most useful part of the post is the honest account of quick fixes and how long each lasted.

Fix	When	How long it bought
Double the cores	October 2025	70 days
Shard the listener by package	February 2026	29 days
Daily restarts	March 2026	Less than a day
Redesign with an in-memory data store	After March	Stable since

Patch 2: sharding by package

The team realised the listener didn’t need a single writer for everything, just “a single writer per package.” Each package’s state became a shard with its own worker. Claude generated the code. It bought 29 days.

Patch 3: restarts

By March, the process hit its memory limit by mid-afternoon most weekdays. Restarting bought less than a day, and the service gradually fell further behind. When it lagged by more than an hour, many job results weren’t recorded. Malhotra is careful to clarify that CI still ran and untested code didn’t ship. The selector simply used stale data, which “mostly this translated into us running tests that were already super flaky or widespread-failing.”

The redesign

The fix was to give the service a data store. Any listener worker can now process any result, append it to a journal in an in-memory store and move on, “stateless and hence, horizontally scalable.” A small consumer rolls the journal into per-test history every few seconds, and the selector reads from it.

It costs more to run but is “much easier to scale and memory profile than a shaky singleton.” The project took three weeks for one engineer. Malhotra notes that “a year ago it would have been closer to a quarter.”

Anthropic’s lessons

Malhotra lists what he’d do differently, and these apply well beyond Anthropic.

1. Plan for exponential load. “Assume your architecture will be at a 25x load within two quarters.”
2. Expect smaller, more frequent PRs. Claude “prefers smaller, more granular PRs,” which means more CI jobs and another reason not to run every test on every PR.
3. Expect activity around the clock. Agents push overnight and at weekends, but load stays bursty because humans still drive many PRs.
4. Instrument everything. “Ensure that the same number of CI jobs coming in equals the same going out.”
5. Keep state out of the process from the start, and avoid running critical services as single instances you can’t measure.

He also notes that the specific techniques, bigger machines, parallelisation, restarts, are not the insight. The insight is that each bought far less time than it would have a year earlier, while a full redesign now takes far less time too.

Spotify’s view: verification is the new constraint

Spotify’s September post, by Tyson Singer, Head of Technology and Platforms, comes at the same problem from the quality side.

Some figures from the post:

• Total merged changes “more than doubled year over year in August, from roughly 8,100 to 17,000.”
• Quality and optimisation work rose from 27% to 31% of the mix.
• In incident retrospectives, Spotify “did not identify AI-authored code as a material direct contributor,” but did see that “the volume of change increased faster than some of our verification controls could adapt.”
• An automated dependency upgrade “passed our checks, but still failed in production.”
• Code complexity and PR size are “creeping up,” and Spotify is deliberately not rewriting its thresholds yet.

The conclusion lines up with Anthropic’s: “AI increased the capacity to produce change. The next constraint became our ability to verify it.”

What this means for QA and platform teams

You probably don’t run CI at Anthropic’s scale. But if your team is adopting coding agents, the shape of the curve will be similar. Here’s how to get ahead of it.

1. Measure your CI load trend now

Track CI jobs per day, PRs per day, tests per run and queue time. Plot them weekly. If the curve is bending upward, assume it will keep bending. Anthropic’s advice to plan for 25x within two quarters is a reasonable stress-test number for your design, even if your growth is slower.

2. Start with simple test selection

You don’t need a custom service on day one. Most ecosystems have built-in options:

• Jest: --changedSince and --findRelatedTests run tests related to changed files. See Jest.
• Vitest: vitest related and --changed do the same. See Vitest.
• Playwright: --only-changed runs test files affected by changes since a git ref. See Playwright.
• Monorepo tools: “affected” commands in build systems select projects based on the dependency graph.

These use static file and dependency relationships. Anthropic’s service adds history: past performance per test. That’s the next step.

3. Always keep a safety net

Test selection trades certainty for speed. Keep a full run somewhere:

• On merge to main, or on a schedule such as nightly.
• Before releases.
• Whenever selection data is stale or the service is degraded.

Anthropic’s post shows why the last point matters. When the listener lagged, selection used stale data. Your pipeline should detect that state and fall back to running more tests, not fewer.

4. Treat the selector as production infrastructure

If test selection decides what gets verified, it’s part of your quality system. Give it:

• An owner. Anthropic notes that early on “ownership was murky.”
• Monitoring and alerts on lag, throughput and errors.
• A “jobs in equals jobs out” check.
• Horizontal scaling and externalised state.

5. Quarantine flaky tests deliberately

Stale selection data in Anthropic’s case meant running tests that were “already super flaky or widespread-failing.” Flaky tests poison both human trust and agent self-verification. A quarantine process with clear exit criteria keeps them out of the signal. We cover approaches, including Reddit’s sequence-based quarantine, in Flaky tests: causes, fixes and Playwright retries.

6. Watch for checks that pass but miss problems

Spotify’s dependency upgrade passed every check and still failed in production. More automated change means more chances for that gap. Strengthen rollout and rollback alongside testing, as Spotify describes: safeguards, expanded rollback capacity and scheduling automated changes during the owning team’s working hours.

7. Give agents the right test set

If agents author many of your PRs, design CI output for them. A focused set of relevant tests with clear failure messages helps an agent self-verify. A wall of unrelated red does the opposite.

A test impact analysis maturity model

Level	What it looks like	When to move on
0. Run everything	Full suite on every PR	CI time or cost becomes painful
1. Static selection	Changed files and dependency graph	Large shared modules trigger most tests
2. History-aware selection	Past failures and flakiness per test	Selection service becomes a bottleneck
3. Scaled selection service	Stateless workers, external store, monitoring	Load keeps growing
4. Agent-aware CI	Targeted results designed for agent self-verification	Continuous tuning

Most teams are at level 0 or 1. The jump to level 2 is where the biggest savings are, and also where you need infrastructure discipline.

A worked example: adding history to test selection

Here’s a simplified, hypothetical design for a team moving from level 1 to level 2. It borrows the listener and selector split from Anthropic’s post, and its stateless lesson, without trying to copy its scale.

Listener. A CI step at the end of every run posts each test’s result to an endpoint: test ID, package, commit, outcome and duration. The endpoint appends it to a queue or journal and returns at once. It holds no state in memory.

Roll-up job. Every few seconds, a consumer reads new journal entries and updates a per-test record: recent pass rate, last failure, average duration and whether the test is quarantined.

Selector. For each PR, the selector:

1. Finds changed packages and the tests that depend on them, as static tools do.
2. Adds tests that failed recently on main, since they’re likely to catch the same problem again.
3. Excludes quarantined tests, but reports them by name so nothing is silently skipped.
4. Adds a small random sample of other tests, to catch gaps in the dependency graph.
5. Checks how fresh its data is. If the roll-up is more than a set number of minutes behind, it selects the full suite and says why.

Monitoring. Dashboards show results received, results processed and lag, matching Anthropic’s advice that jobs in should equal jobs out.

This design is intentionally boring. Every part can be scaled out by adding workers, and the staleness check turns a degraded selector into slower CI rather than missed tests.

Questions to ask a test selection vendor

Anthropic notes that “there is a category of vendors with offerings in this area.” If you’re buying rather than building, ask:

• What signals does selection use? File changes, coverage, history, or a model? Can you see why each test was chosen or skipped?
• How does it behave when data is stale? Does it fall back to running more tests?
• What load has it been tested at? Ask for figures at many times your current CI volume.
• How does it handle flaky tests? Are they quarantined openly or quietly skipped?
• Can you audit misses? When a regression reaches main, can you tell whether selection skipped the test that would have caught it?
• How do agents consume the output? Is there a clear, machine-readable list of what ran and why?

The answers matter more than headline savings. A selector that saves most of your CI time but quietly skips the wrong test is a liability.

Risks of test impact analysis

• Missed regressions when dependencies are dynamic, such as configuration, reflection or data-driven behaviour.
• Stale data when the history service lags.
• False confidence if teams forget that a passing PR ran only a subset.
• Hidden flakiness if selection quietly skips unstable tests without a quarantine record.

Mitigate them with scheduled full runs, clear reporting of what was skipped, and mutation testing to check that the tests you do run actually catch faults.

Frequently asked questions

What is test impact analysis?

Test impact analysis, or test selection, chooses which tests to run for a change based on what changed and, in more advanced systems, on each test’s history. It shortens CI runs compared with running the full suite on every pull request.

Why is agentic coding straining CI?

Agents produce more and smaller pull requests, often around the clock. Anthropic reported a 25x increase in CI jobs over six months as code output and test counts grew.

How did Anthropic fix its test selection service?

After three short-lived patches, it redesigned the service so stateless listener workers append results to a journal in an in-memory data store, with a separate consumer building per-test history. The service has been stable since.

Is test impact analysis safe?

It is safe when combined with scheduled full runs, monitoring for stale data and clear reporting of skipped tests. Without those, it can miss regressions.

Can I do test impact analysis without a custom service?

Yes. Tools such as Jest, Vitest and Playwright support running tests related to changed files, which is a good first step before history-aware selection.

Further reading on CheckOps

• Flaky tests: causes, fixes and Playwright retries
• Mutation testing for AI-written tests
• Meta’s JiTTests and just-in-time tests
• Agentic vs deterministic E2E tests: lessons from Slack
• CrowdStrike outage lessons for release quality