Flaky Tests: Causes, Fixes and Playwright Retries

Last updated 30 October 2026, including Playwright 1.62 isolated retries and 1.63 test locks.

A flaky test is one that both passes and fails on the same code. It’s one of the most expensive problems in test automation, because it wastes CI time and, worse, teaches people to ignore red builds. Once a team starts saying “just rerun it”, real bugs slip through.

This guide explains what causes flaky tests, how to find them, how to fix the common types, when to quarantine, and how to use Playwright’s retry features without hiding problems. Most of the advice applies to any framework, with Playwright examples where they help.

How common are flaky tests?

Google published one of the best-known datasets on this. In a May 2016 post on the Google Testing Blog, John Micco wrote that:

• about 1.5% of all test runs at Google reported a flaky result,
• almost 16% of tests had some level of flakiness, and
• about 84% of pass-to-fail transitions observed in post-submit CI involved a flaky test.

These figures are from 2016 and from one company’s very large test corpus, so don’t treat them as a benchmark for your team. They’re still useful because they show two things: even well-resourced teams can’t remove flakiness entirely, and flaky failures can swamp real ones. Google noted the insertion rate of flaky tests was about the same as the fix rate.

The same post defined a flaky result as “a test that exhibits both a passing and a failing result with the same code”, which is the definition this guide uses.

The common causes of flaky tests

Most flakiness comes from a short list of causes.

1. Timing and waiting

The most common cause in UI tests. The test acts before the page is ready, or asserts before data has loaded. Fixed sleeps, such as waitForTimeout(2000), make this worse, because they’re too short on slow machines and waste time on fast ones.

2. Shared state between tests

Tests that depend on data created by other tests, share a user account, or leave records behind can fail depending on the order they run in, or when they run in parallel.

3. Test data and environment

Shared staging environments, data that other people change, expiring tokens and seeded data that drifts all cause intermittent failures.

4. Concurrency and race conditions

Sometimes the bug is real. A race condition in the application can make a test fail only occasionally. These are the most important flaky tests to investigate, because they may be the only signal of a production bug.

5. External dependencies

Third-party APIs, payment sandboxes, email services and feature flag services can be slow or unavailable.

6. Non-deterministic code

Random values, current dates and times, time zones, locale settings and unordered collections can all produce different results between runs.

7. Infrastructure

Resource-starved CI runners, network blips, browser crashes and container limits can make healthy tests fail.

8. Brittle selectors

Selectors tied to layout, generated class names or text that changes can break intermittently, for example when A/B tests or feature flags render different markup.

How to find flaky tests

You can’t fix what you can’t see. A few practical methods:

Track results over time. Record every test result in CI, including retries. A test that fails and then passes on retry is flaky by definition. Playwright’s reports mark such tests as “flaky”.

Rerun suspicious tests many times. Playwright’s --repeat-each option runs each test N times:

npx playwright test tests/checkout.spec.ts --repeat-each=50 --workers=4

Running in parallel with several workers also helps expose shared-state problems.

Rerun only failures. --last-failed reruns just the failures from the last run, which is handy when investigating.

Make flakiness visible in CI. Playwright’s --fail-on-flaky-tests option fails the run if any test is flagged as flaky. Some teams use it on main-branch runs to stop flakiness accumulating quietly.

Use a dashboard. Test reporting tools such as Currents, Allure Report and ReportPortal can track flaky rates per test over time.

How to fix the common types

Fix timing problems with web-first assertions

Replace sleeps and manual polling with assertions that wait automatically.

// Flaky: fixed sleep, then a one-off check
await page.waitForTimeout(2000);
expect(await page.locator('.toast').isVisible()).toBe(true);

// Stable: web-first assertion that retries until timeout
await expect(page.getByRole('status')).toHaveText('Saved');

Playwright’s actions also auto-wait for elements to be actionable, so avoid adding extra waits before clicks.

Wait for the right thing

If a test depends on a network response, wait for that response rather than a fixed time:

const responsePromise = page.waitForResponse('**/api/orders');
await page.getByRole('button', { name: 'Place order' }).click();
await responsePromise;

Use resilient locators

Prefer role, label and test ID locators over CSS chains:

await page.getByRole('button', { name: 'Checkout' }).click();
await page.getByLabel('Email').fill('user@example.com');
await page.getByTestId('order-total');

Playwright 1.63 added locator.visible(), which matches only visible elements and is the recommended replacement for the :visible CSS pseudo-class. That helps when hidden duplicates of an element exist, for example in responsive layouts.

Isolate test data

Give each test its own data. Create users, orders or records through the API in a fixture, and clean up afterwards:

import { test as base } from '@playwright/test';

export const test = base.extend<{ user: { email: string } }>({
  user: async ({ request }, use) => {
    const email = `user-${Date.now()}-${Math.random().toString(36).slice(2)}@example.com`;
    await request.post('/api/test-users', { data: { email } });
    await use({ email });
    await request.delete(`/api/test-users/${encodeURIComponent(email)}`);
  },
});

Each test gets a unique user, so parallel tests don’t collide.

Control time, randomness and locale

Pin the time zone and locale in your config, and control the clock where a feature depends on dates. Playwright’s Clock API (page.clock) lets you set or fast-forward time in the browser. Seed random number generators in the application under test when you can.

// playwright.config.ts
export default defineConfig({
  use: {
    timezoneId: 'UTC',
    locale: 'en-GB',
  },
});

Mock unreliable external services

For tests that aren’t specifically about a third-party integration, stub it. Playwright’s page.route() can return a fixed response. For API-level mocks, tools like WireMock or Mockoon work well. Keep a smaller set of contract or integration tests for the real service.

Investigate real race conditions

When a test fails only under load or in parallel, don’t dismiss it. Use traces to see exactly what happened. Playwright’s Trace Viewer shows each action, network request, console message and DOM snapshot. Since 1.63, traces can also record aria and screen snapshots on every action.

// playwright.config.ts
export default defineConfig({
  use: { trace: 'on-first-retry' },
});

Playwright retries: helpful, but handle with care

Retries are a safety net, not a fix. Used well, they keep CI moving while you fix flaky tests. Used badly, they hide real bugs.

Basic retries

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
});

A test that fails then passes on retry is reported as “flaky”, not “passed”. That label is valuable data. Don’t throw it away.

You can check the retry number inside a test with test.info().retry, for example to clear a cache before retrying.

Isolated retries (Playwright 1.62)

Playwright 1.62 added retryStrategy. The default, 'immediate', retries as soon as a worker is free. The 'isolated' strategy runs all retries at the end, one by one in a single worker, to minimise interference with the rest of the suite.

export default defineConfig({
  retries: 2,
  retryStrategy: 'isolated',
});

This is useful for diagnosis. If a test fails in the parallel run but passes when retried in isolation, the cause is probably shared state, resource contention or load, not the test logic itself.

Test locks (Playwright 1.63)

Some tests really do share a resource, such as a global setting or an external account. Before 1.63, teams often made whole files serial or reduced workers. Playwright 1.63 added test locks: tests that declare the same lock name never run at the same time, across files, workers and projects, while everything else stays parallel.

test('update account settings', { lock: 'account-settings' }, async ({ page }) => {
  // never runs at the same time as other tests holding 'account-settings'
});

A test can hold several locks, and test.describe() accepts a lock for a whole group. Use locks for genuine shared resources. Use isolated test data for everything else.

Retry rules worth adopting

1. Never retry locally by default. Developers should see flaky failures.
2. Keep CI retries low, typically one or two.
3. Report flaky tests separately and review them every week.
4. Set a budget. For example, “no more than N flaky tests on main” or “flaky tests fixed within one sprint”.
5. Fail on flakiness where it matters, using --fail-on-flaky-tests on release branches or nightly runs.
6. Record traces on retry so every flaky failure comes with evidence.

Quarantine: when and how

Sometimes a flaky test can’t be fixed quickly, and it’s blocking everyone. Quarantining moves it out of the blocking path while keeping it visible.

A simple approach in Playwright is a tag plus a separate project or CI job:

test('exports large report', { tag: '@quarantine' }, async ({ page }) => {
  // ...
});

# Blocking job
npx playwright test --grep-invert @quarantine
# Non-blocking job, results still tracked
npx playwright test --grep @quarantine

Rules that stop quarantine becoming a graveyard:

• Every quarantined test has an owner and a ticket.
• Quarantine has a time limit, after which the test is fixed or deleted.
• Quarantined tests still run and report, just without blocking.
• The quarantine list is reviewed regularly.

How Reddit automates quarantine

Large teams automate this. Reddit engineer Abinodh Thomas described a sequence-based dynamic test quarantine system in From Fragile to Agile Part II, featured in Software Testing Weekly on 3 February 2026. The system is built from services written in Go, including a Sequencer, Feeder, Validator and a Quarantine Phase Store. As the post describes it, quarantine decisions follow the sequence of merged commits, quarantine phases are closed through JIRA, and there’s an in-repository fallback for when the service can’t be reached. Read the post for the full design. The general lesson is that quarantine works best when it’s automatic, tracked and time-limited.

Measuring flakiness

To manage flakiness, measure it consistently. Useful numbers:

Metric	How to calculate	Why it matters
Flaky rate per test	Runs flagged flaky ÷ total runs, over a window such as 30 days	Finds the worst offenders
Suite flaky rate	Runs with at least one flaky test ÷ total runs	Shows how often developers are disrupted
Retry cost	Extra minutes spent on retries	Turns flakiness into CI cost
Time to fix	Days from first flake to fix or deletion	Shows whether your process works
Quarantine size	Number of quarantined tests	Warns when quarantine becomes a graveyard

Fix the tests with the highest flaky rate on the most important paths first. A flaky checkout test matters more than a flaky test for a rarely used settings page.

Flaky tests in Cypress and other frameworks

The causes are the same in every framework, and so are most fixes. A few framework notes:

• Cypress supports test retries through its retries configuration, with separate values for cypress run and cypress open. Cypress 16 also made cookie and storage commands, such as cy.getCookie() and cy.getAllLocalStorage(), into retrying queries, which removes a source of timing-related flakiness. See our Playwright vs Cypress comparison.
• Jest and Vitest unit tests are flaky less often, but shared module state, timers and real network calls cause most of the problems. Use fake timers and mocks.
• Mobile tests with Appium or Espresso add device and emulator variability. Pin device images and reset app state between tests.

Preventing flaky tests

Prevention is cheaper than cleanup.

• Review tests like production code, with extra attention to waits, data and shared state.
• Run new tests many times before merging, for example with --repeat-each=20 in a CI job for changed test files.
• Keep tests independent. Each test should set up its own state and work in any order.
• Prefer API setup over UI setup for preconditions, which is faster and less brittle.
• Keep end-to-end suites lean. Push checks down to unit and API tests where possible.
• Watch AI-generated tests closely. Coding agents can produce tests that pass for the wrong reasons or rely on timing. Our mutation testing guide shows how to check whether tests actually catch bugs.

Flaky tests and AI agents

Agent-driven tests add a new kind of variability. In Slack’s study, only about 20% of agent runs followed exactly the same sequence of actions. That isn’t flakiness in the classic sense, since many paths were valid, but it means you need pass-rate thresholds and run logs, not single pass/fail results. See Agentic vs deterministic E2E tests.

A flaky test triage checklist

When a test flakes, work through these questions:

1. Does it fail locally with --repeat-each?
2. Does it fail only in parallel, or also with --workers=1?
3. Does it pass when retried in isolation with retryStrategy: 'isolated'?
4. What does the trace show at the failing step?
5. Does it depend on shared data, time, locale or an external service?
6. Could this be a real race condition in the app?
7. Who owns the fix, and by when?

Frequently asked questions

What is a flaky test?

A test that both passes and fails on the same code without any change. Google’s Testing Blog uses this definition.

What causes flaky tests most often?

Timing and waiting problems are the most common in UI tests, followed by shared state, unstable test data and environments, external dependencies and real race conditions.

Should I use retries in Playwright?

Yes, in CI, with a low count and separate reporting of flaky tests. Don’t use retries to hide failures. Review flaky tests regularly and fix them.

What is retryStrategy ‘isolated’ in Playwright?

Added in Playwright 1.62, it runs all retries at the end of the run, one by one in a single worker, to reduce interference from the rest of the suite.

What are Playwright test locks?

Added in Playwright 1.63, they let tests declare a named lock. Tests sharing a lock never run concurrently, while the rest of the suite stays parallel.

Should I delete flaky tests?

Sometimes. If a test is low value, hard to fix and duplicated by other checks, deleting it may be the right call. Otherwise, quarantine it with an owner and deadline, then fix it.

Further reading on CheckOps

• Playwright vs Cypress in 2026
• Test impact analysis for agentic coding
• Playwright Test Agents and self-healing tests
• Mutation testing for AI-written tests
• Tools: Playwright, Currents, Allure Report, WireMock