Meta JiTTests: Just-in-Time Tests for Agentic Code

On 11 February 2026, Meta’s engineering blog published a post with a deliberately provocative title: “The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It”. The author was Mark Harman, one of the best-known names in search-based software engineering. The post introduced Just-in-Time Tests, or JiTTests, and pointed to a paper on arXiv, Just-in-Time Catching Test Generation at Meta, submitted on 30 January 2026 to the FSE 2026 industry track.

Strip away the headline, and the idea is both radical and very practical. Instead of maintaining a permanent suite of tests that must anticipate every future change, generate tests for each change, at the moment it is proposed, with the explicit goal of making them fail if the change introduced a bug.

This essay explains how Meta’s Catching JiTTests work, what the paper actually reports, where the approach is strong and weak, and what an ordinary QA team can take from it. It follows our earlier essays on AI coding agents writing tests and LLM testing and evals.

What are JiTTests?

Meta’s February post defines Just-in-Time Tests as “a fundamentally novel approach to testing where tests are automatically generated by large language models (LLMs) on the fly to catch bugs … just-in-time before the code lands into production.”

The specific variant Meta describes is the Catching JiTTest, which “focuses specifically on finding regressions introduced by a code change.”

Hardening tests vs catching tests

The paper draws a distinction that every tester should know.

	Hardening test	Catching test
Goal	Protect existing behaviour from future changes	Reveal a bug in the current change
Passes at generation time?	Yes, by design	No. It is meant to fail if the change is buggy
Lives in the codebase?	Yes, permanently	No, generated per change
Maintenance	Ongoing	None
Who reviews it?	Engineers review the test code	Engineers review only reported catches

Most automated test generation, including most AI unit test generators, produces hardening tests. They capture what the code does today, then sit in the repository forever. Catching tests flip that. They exist only to answer one question: did this diff break something?

Why Meta says traditional testing is struggling

Harman’s argument is that agentic development has changed the economics of tests. Under the traditional model, engineers “face the challenge of needing to check the behavior, not only of the current code, but all possible future changes”. The result, in his words, is “tests that don’t catch anything, or when they do, it’s a false positive.”

When AI agents write more code, more quickly, two costs scale with them: writing and maintaining tests, and investigating false positive failures. If tests are generated per change and discarded afterwards, the maintenance cost disappears. The remaining problem is false positives, and much of the paper is about that.

How Catching JiTTests work

Meta’s February post lists six steps:

1. New code lands in the codebase, in the sense of a proposed change, or diff.
2. The system infers the intention of the code change.
3. It creates mutants, “code versions with faults deliberately inserted”, to simulate what could go wrong.
4. It generates and runs tests to catch those faults.
5. “Ensembles of rule-based and LLM-based assessors” focus the signal on true positive failures.
6. Engineers receive “clear, relevant reports about unexpected changes right when it matters most.”

Mutants as a guide for test generation

Mutation testing has been around for decades. You insert small faults into code, such as flipping a comparison, and check whether your tests notice. If they don’t, your tests are weak. Meta’s twist is to use mutants as a guide for an LLM generating tests. Its earlier work on LLM-powered bug catchers in February 2025 and mutation testing for compliance in September 2025 laid the foundations.

For a catching test, the logic is: if an LLM can describe plausible ways the change could go wrong, and generate tests that would detect those faults, then running those tests on the real change may reveal a real fault.

Filtering false positives with assessors

A test that fails on a diff is only a “candidate catch”. It might reveal a bug, or the change might be intentional and the test simply wrong about the intent. The paper uses both rule-based and LLM-based assessors to rank and filter candidates before any human sees them.

What the paper reports

The arXiv abstract gives the core numbers. These are Meta’s own results, on Meta’s backend systems, which the abstract describes as “hundreds of millions of line of code”.

Finding	Value reported
Generated tests analysed	22,126
Candidate catch generation, code-change-aware vs hardening tests	4x better
Candidate catch generation vs coincidentally failing tests	20x better
Reduction in human review load from assessors	70%
Candidate catches reported to engineers	41
Confirmed true positives	8
Of those, would have led to serious failures	4

The abstract also notes an interesting statistical result: “human-accepted code changes are assessed to have significantly more false positives, while human-rejected changes have significantly more true positives.” In other words, the assessors’ judgement correlated with what human reviewers later decided about the code.

How to read these numbers

Two things are worth saying plainly.

First, 8 confirmed bugs from 41 reported catches is a modest absolute number. This was an initial deployment, not a claim that JiTTests catch every regression.

Second, 4 of those 8 would have caused serious failures. In a codebase that size, preventing even a handful of serious incidents before they land can justify a lot of compute. The question for your team is not whether JiTTests are perfect, but whether the cost per serious bug caught is better than your alternatives.

A worked example: what a catching test looks like

To make the idea concrete, here is a small, hypothetical example. It is not from Meta’s paper; it simply shows the shape of the technique.

Suppose a pull request titled “Apply loyalty discount only to orders over 50” changes this function:

def final_price(total, is_loyal):
    if is_loyal and total > 50:
        return round(total * 0.9, 2)
    return total

An intent-aware generator would first restate the intent: loyal customers get 10% off when the order total is above 50, and nobody else gets a discount. Then it would list risks, the ways an implementation of that intent could go wrong:

• The boundary is wrong, for example >= instead of >, or the reverse.
• Non-loyal customers get the discount.
• The discount is applied twice or at the wrong rate.
• Rounding changes the price for totals that should not change.

Each risk becomes a mutant, and the generator writes tests that tell the mutant apart from the parent version of the code:

def test_boundary_total_of_50_gets_no_discount():
    assert final_price(50, True) == 50

def test_non_loyal_customer_never_discounted():
    assert final_price(120, False) == 120

def test_loyal_customer_over_threshold_gets_ten_percent():
    assert final_price(100, True) == 90.0

Those tests are then run against the actual diff. If the developer had typed >= by mistake, the first test fails, and that failure is a candidate catch. An assessor then asks whether the failure reflects a real bug or a misunderstanding of intent. Here the pull request title says “over 50”, so the failure is likely a true positive, and the engineer receives a short report rather than a test file to review.

Notice what did not happen. Nobody added these tests to the repository, and nobody will maintain them. If the behaviour matters long term, a human can choose to promote one into the durable suite.

How this differs from AI code review

It is worth separating JiTTests from the AI code review bots many teams adopted in 2025. A review bot reads the diff and comments on it, using patterns it has learned. A catching test runs the code and shows a concrete difference in behaviour. The first is an opinion; the second is evidence. Meta’s October follow-up makes the same point, describing Catching JiTTests as “a fundamentally different approach” that “catches classes of bugs that LLM- or agent-based code reviews might not.” In practice the two work well together: the reviewer flags risky areas, and the generated tests try to prove or disprove the risk.

Strengths of the JiTTest approach

No test maintenance

Because catching tests are generated per change and discarded, nobody has to keep them up to date. This directly attacks the biggest hidden cost of large automated suites.

Tailored to intent

The tests are built around what the change is trying to do. That makes them less likely to break for unrelated reasons, and more likely to probe the risky parts of the change.

Humans review outcomes, not test code

Meta’s post says Catching JiTTests “only require human review when a bug is actually caught”. Engineers are asked a simpler question: was this behaviour change intended?

Complements code review

An LLM code reviewer reads code and comments. A catching test executes code and demonstrates a concrete behavioural difference. Those are different kinds of evidence, and they catch different classes of bugs.

Weaknesses and open questions

Compute cost

Generating mutants and tests for every change, then running assessors, is expensive. Meta can run it at scale; many teams cannot, at least not for every pull request.

It does not replace regression suites

Catching tests look for bugs in a change. They do not protect a critical user journey from slow drift across many changes, and they do not express agreed business rules the way a reviewed test suite does. You still need deterministic regression checks for the behaviours that matter most, as we argue in testing vs checking.

Intent inference can be wrong

If the system misunderstands what a change is for, it may generate mutants and tests that miss the real risk. Good pull request titles and descriptions become test inputs, not just documentation.

Backend focus

The paper describes backend systems. Applying the same approach to UI, mobile or end-to-end flows is harder, because tests are slower and environments are less controllable.

What QA teams can borrow today

You do not need Meta’s infrastructure to use some of these ideas.

1. Ask your AI tools for catching tests

When you use a coding agent on a pull request, ask it explicitly to write tests that would fail if the change were wrong, not just tests that pass. Then check that at least one generated test fails on a deliberately broken version of the code.

2. Use mutation testing to check AI-written tests

Tools such as Stryker for JavaScript and PIT for Java show whether your tests detect injected faults. We go deeper on this in mutation testing for AI-written tests.

3. Write intent into pull requests

If intent drives test generation, then a clear “what should change and what must not change” section in the pull request description improves both human and machine review.

4. Separate disposable tests from durable tests

Not every test needs to live forever. Some tests exist to verify one change. Be explicit about which tests are durable regression checks and which are throwaway verification, and keep your durable suite lean.

5. Measure true positives

Track how many AI-reported failures turn out to be real bugs. If your AI testing tools mostly report noise, engineers will learn to ignore them, and you will lose the benefit.

Update, October 2026: JiTTests at full scale

On 29 October 2026, Meta published a follow-up, Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale, by Sreeja Thummala, Sophie Zeng, Mark Harman, Weini Liu and Eric Liu. It says Catching JiTTests “are no longer a research prototype; they’re part of Meta’s infrastructure”, and adds detail on how the system now works:

• Targeting: it runs overnight on spare capacity, focusing on the previous day’s diffs deemed “risky” by a mechanism similar to Meta’s Diff Risk Score.
• Two workflows: a “dodgy diff” workflow that treats each diff as if it were a mutant, and an “intent-aware” workflow that infers the diff’s intent, describes risks, and builds mutants from them.
• Assessment: an LLM-as-Judge ensemble of several models, combined with a deterministic rule-based assessor called RubFake.
• Human-on-the-loop: engineers “set criteria and review outcomes at an aggregate level rather than supervising every output”.

Meta reports that 589 engineers, 64% of its outreach, have engaged with Catching JiTTests, that they report a 60% true-positive rate, and that in the first half of 2026 the system “played a key role in catching 74 serious (SEV) privacy bugs at the diff stage”, lowering privacy SEVs by 20% over that period. As before, these are Meta’s own figures.

What this means for your team

JiTTests are a strong signal about where testing is heading in the agentic era. When code changes faster than humans can write and maintain tests, testing shifts from “own a big suite” to “generate evidence for each change, and keep a small, trusted core”.

For most teams in 2026, the practical steps are modest: ask AI tools for failing tests, verify them with mutation testing, write intent into pull requests, and measure whether automated findings are real. The role of QA becomes setting the criteria, judging the outcomes and owning the durable checks that encode what the business cares about.

Frequently asked questions

What does JiTTest stand for?

Just-in-Time Test. Meta uses the term for tests generated by LLMs on the fly for a specific code change, just before it lands.

What is a Catching JiTTest?

A test generated to reveal a regression in a particular code change. Unlike a hardening test, it is meant to fail if the change introduced a bug, and it is not kept in the codebase afterwards.

How many bugs did Meta’s JiTTests find?

In the paper’s initial deployment, 41 candidate catches were reported to engineers, 8 were confirmed true positives and 4 of those would have caused serious failures. Meta’s October 2026 follow-up reports 74 serious privacy bugs caught at the diff stage in the first half of 2026.

Do JiTTests replace unit tests?

No. They complement a durable regression suite by focusing on bugs introduced by each change. You still need reviewed tests for critical behaviour.

How do JiTTests use mutation testing?

The system creates mutants, versions of the code with deliberately inserted faults that represent plausible mistakes, and generates tests that can detect them. Those tests are then run on the real change to look for real faults.

Further reading on CheckOps

• Mutation testing for AI-written tests
• AI coding agents writing tests: Claude Code and Copilot
• LLM testing, evals and vibe coding quality
• Testing vs checking: why your software needs both
• Shift left, shift right: testing and observability