The CheckOps Engineer home

Essay

Playwright vs Cypress vs Selenium: The BiDi Shift

Playwright vs Cypress vs Selenium in late 2024: WebDriver BiDi, download trends, architecture trade-offs and how to pick a web test automation framework.

15 September 2024 · 12 min · Irfan Ahmad

For most of the last decade, choosing a web test automation framework meant choosing a protocol, often without realising it. Selenium spoke W3C WebDriver. Cypress ran inside the browser. Playwright and Puppeteer drove browsers over the Chrome DevTools Protocol (CDP) or patched equivalents. In 2024 those foundations started to converge, and the “Playwright vs Cypress” argument took on a new dimension.

This essay covers mid-August to mid-September 2024. The news is mostly releases and signals: Selenium 4.24’s BiDi work, Playwright 1.47, Firefox retiring CDP, and the download chart that had everyone talking. The deeper theme is how to choose a framework in a year when the protocol layer is becoming a W3C standard.

The signal everyone screenshotted

In June 2024 Playwright overtook Cypress in weekly npm downloads for the first time. Download counts are a noisy proxy. They include CI installs, transitive dependencies and bots, and they say nothing about satisfaction. But the trend line had been steady for two years, and by late summer the crossover was the context for every framework discussion on Reddit, in Software Testing Weekly and in conference hallways.

It would be wrong to read this as “Cypress is dying”. Cypress has a huge installed base, an excellent developer experience for component and front-end testing, and a commercial cloud that many teams rely on. What the chart does show is that new projects increasingly start on Playwright, and that matters for hiring, community answers and plugin ecosystems.

What changed under the hood: WebDriver BiDi

WebDriver BiDi is a W3C protocol that adds what classic WebDriver lacked: a bidirectional, event-driven channel. Classic WebDriver is request/response over HTTP. You ask the browser to click and wait for an answer. CDP gave Chromium-based tools events such as console logs, network requests and DOM mutations, but it is a Chromium-specific, unstable-by-design protocol.

BiDi standardises those capabilities across browsers. The past few weeks showed how far it has come:

For Selenium users this is the most important development in years. Many features people moved to Playwright for, such as listening to console errors, intercepting network calls and reacting to DOM changes, are arriving in Selenium through a standard protocol rather than vendor-specific hacks.

Here is what listening to console messages looks like with Selenium’s BiDi-backed API in Java:

try (LogInspector inspector = new LogInspector(driver)) {
    List<ConsoleLogEntry> logs = new ArrayList<>();
    inspector.onConsoleEntry(logs::add);
    driver.get("https://example.test/checkout");
    // ... drive the flow ...
    assertThat(logs).noneMatch(e -> e.getLevel() == LogLevel.ERROR);
}

And the equivalent in Playwright, which has had this for years over its own protocol layer:

const errors: string[] = [];
page.on('console', msg => { if (msg.type() === 'error') errors.push(msg.text()); });
await page.goto('https://example.test/checkout');
// ... drive the flow ...
expect(errors).toEqual([]);

The ergonomics differ, but the gap in capability is narrowing.

Playwright 1.47: small release, telling details

Playwright 1.47 landed on 5 September. Its headline features are modest. The UI mode network tab gained filtering by asset type and URL. A --tsconfig flag lets one TypeScript config apply to every imported file. The release pattern is the interesting part. Playwright now ships roughly monthly, and each release polishes the debugging experience: trace viewer, UI mode, HTML reporter. Last month’s 1.46 added --only-changed for diff-based test selection, which we covered in the CrowdStrike release-quality essay.

That steady investment in tooling around the runner, not only the browser API, is a big part of why Playwright is winning new projects.

Architecture, not features, should decide

Feature checklists for these three frameworks are mostly a tie in 2024. Each can do almost anything with enough plugins. The real differences are architectural, and they decide what is easy.

Dimension Playwright Cypress Selenium 4
Execution model Out-of-process, drives browser over protocol In-browser, test code runs alongside app Out-of-process, W3C WebDriver (+ BiDi)
Languages JS/TS, Python, Java, .NET JS/TS Java, Python, C#, JS, Ruby, Kotlin
Browsers Chromium, Firefox, WebKit builds Chromium family, Firefox, WebKit (experimental) All major browsers via vendor drivers
Multi-tab / multi-origin First-class Constrained (cy.origin helps) First-class
Parallelism Built into runner (free) Via Cypress Cloud or third-party Selenium Grid / cloud vendors
Auto-waiting Built-in actionability checks Built-in retry-ability Explicit/implicit waits (you design it)
Debugging Trace viewer, UI mode, codegen Time-travel runner, excellent DX Depends on stack; BiDi adds events
Component testing Experimental Mature Not a focus
Mobile Emulation only Viewport emulation Real devices via Appium
Standards-based Own protocol layer Own architecture Yes (W3C)

When Playwright is the right call

  • New JS/TS or Python projects that need multiple browsers, multiple tabs, iframes or cross-origin flows.
  • Teams that want free, built-in parallelism and sharding without a paid cloud.
  • Organisations that value the trace viewer for debugging CI failures. A trace file attached to a failed run cuts triage time dramatically.

When Cypress is the right call

  • Front-end-heavy teams where developers write most tests and value the interactive runner.
  • Component testing in React, Vue, Angular or Svelte, where Cypress is mature and pleasant.
  • Teams already invested in Cypress Cloud’s analytics, flake detection and test replay.

When Selenium is the right call

  • Polyglot organisations, especially Java and C# shops, with large existing suites and Grid infrastructure.
  • Anything that has to share a protocol with real-device mobile testing through Appium.
  • Regulated environments that prefer W3C standards and multi-vendor support. BiDi makes Selenium more competitive here, not less.

Beyond the browser: API and component layers

An end-to-end framework is rarely used only for end-to-end tests. In practice it becomes the place teams also write API checks and component tests, and the three frameworks support that very differently.

Playwright includes an API testing client, the request fixture, that shares cookies and storage state with browser contexts. That makes a common pattern trivial: seed state through the API, then verify through the UI.

test('order appears in history', async ({ page, request }) => {
  const res = await request.post('/api/orders', { data: { sku: 'ABC-1', qty: 2 } });
  expect(res.ok()).toBeTruthy();
  await page.goto('/account/orders');
  await expect(page.getByRole('row', { name: /ABC-1/ })).toBeVisible();
});

Cypress offers cy.request for the same job, and its component testing mode is the most mature of the three. For front-end teams it can replace a separate component test runner entirely. Selenium has no API client at all. Java teams typically pair it with REST Assured, and others with whatever HTTP library their language favours. That’s more assembly, but it also means your API tests aren’t tied to your UI framework’s lifecycle, which some architects prefer.

The broader point is that the test pyramid hasn’t gone away because end-to-end tools got better. Fast unit tests in Vitest, Jest or pytest, contract tests with Pact, a thinner layer of API tests, and a focused set of end-to-end journeys is still the shape that keeps pipelines fast. Choose the end-to-end framework that fits your top layer. Don’t let it swallow the layers underneath.

Migration: when to move and when to stay

Most teams reading framework comparisons aren’t starting from zero. They’re asking whether to migrate. Our view is that migration pays off when the current framework is the bottleneck, not when another framework is fashionable.

Signals that migration is worth it:

  • Flakiness rooted in the framework’s architecture, such as cross-origin limits or iframe handling, rather than in your test design.
  • CI cost or duration dominated by a lack of parallelism you can’t fix within the current stack.
  • Hiring and onboarding friction because your stack has a shrinking community.

Signals that it isn’t:

  • Flakiness from poor waits, shared test data or environment instability. That follows you to any framework.
  • A suite in the thousands of tests with good stability and owners who know it well.
  • No plan to retire the old suite. Running two frameworks forever is the worst outcome.

If you do migrate, do it incrementally. Run the new framework for new tests, port high-value flows first, and track flake rate and runtime per framework in the same dashboard. A tool like Currents or ReportPortal helps here, as does a shared Allure Report setup.

A minimal comparison you can run yourself

Benchmarks in blog posts are usually unrepresentative. Run your own. Pick three real flows from your app: login, a search-and-filter page and a checkout or form submission. Implement them in each candidate framework, keeping test design identical. Then measure:

metric                       how
---------------------------  -------------------------------------------
cold run time (CI)           median of 10 runs on your CI runner
flake rate                   failures across 50 reruns with no code change
lines of test code           cloc on the test folder
time to diagnose a failure   inject a bug; time from red build to root cause

The last metric is the one teams forget, and it’s usually where frameworks differ most. Playwright’s trace viewer and Cypress’s time-travel runner both do well. A bare Selenium setup often needs extra investment in screenshots, logs and video.

Locators: where flakiness is born

Ask any team why their suite is flaky and the honest answer usually involves locators. The three frameworks now push you toward the same good practice, user-facing and accessibility-based locators, but they get there differently.

Playwright’s recommended locators are role- and label-based, with auto-waiting built into every action:

await page.getByRole('button', { name: 'Place order' }).click();
await expect(page.getByRole('status')).toHaveText('Order confirmed');

Cypress gets close with Testing Library’s commands, which most teams add as a plugin:

cy.findByRole('button', { name: 'Place order' }).click();
cy.findByRole('status').should('have.text', 'Order confirmed');

Selenium leaves the strategy to you. Many mature Selenium suites end up with a thin helper layer that standardises on data-testid attributes and explicit waits:

By placeOrder = By.cssSelector("[data-testid='place-order']");
new WebDriverWait(driver, Duration.ofSeconds(10))
    .until(ExpectedConditions.elementToBeClickable(placeOrder)).click();

None of these is wrong, but the defaults matter. When the default is a role-based locator with built-in waiting, junior engineers and AI assistants write stable tests without trying. When the default is “write your own wait”, stability depends on discipline. That’s a big part of why Selenium suites have a reputation for flakiness that the protocol doesn’t deserve.

Role-based locators have a bonus side effect. They fail when your accessibility tree is broken, so your functional suite doubles as a light accessibility check. Pair it with axe-core for real coverage, but don’t underestimate how many labelling bugs role-based locators surface on their own.

Total cost of ownership, honestly

Framework comparisons rarely talk about money, but money is often what decides. A rough model for a team running a few thousand end-to-end tests per day:

Cost line Playwright Cypress Selenium
Licence Free (Apache-2.0) Free app (MIT); Cloud is paid Free (Apache-2.0)
Parallelism Free sharding in runner Cloud orchestration or DIY Grid (self-host) or vendor
Dashboards / flake analytics Third-party or DIY Built into Cypress Cloud Third-party or DIY
Infra to run browsers CI runners CI runners CI runners + Grid nodes
Real-device / real-Safari Cloud vendor Cloud vendor Cloud vendor or device lab
Engineering time Low setup, low maintenance Low setup; workarounds for multi-origin Higher setup; mature patterns

The pattern here is that “free” frameworks still cost money in CI minutes, dashboards and engineers’ time. Cypress concentrates its cost in a commercial cloud that delivers real value. Playwright and Selenium spread it across infrastructure and tooling you assemble yourself. Neither model is better in the abstract. What matters is which one your organisation can sustain without the suite rotting.

A 90-day evaluation plan

If your team is genuinely undecided, avoid the classic mistake of a two-day spike followed by a months-long argument. A structured evaluation looks like this:

  1. Weeks 1–2: pick flows and criteria. Choose three to five representative flows, including one that crosses origins or tabs and one that’s notoriously flaky today. Agree on weighted criteria up front: stability, runtime, debuggability, language fit and cost.
  2. Weeks 3–6: build in parallel. Two engineers implement the same flows in two candidates, using each framework’s recommended patterns rather than a lowest-common-denominator abstraction.
  3. Weeks 7–10: run in CI. Run both suites on every merge for a month. Record pass rate, rerun rate, median and p95 duration, and time to diagnose every failure.
  4. Weeks 11–13: decide and document. Score against the criteria, write an architecture decision record, and plan the migration or the decision not to migrate.

The month in CI is the critical part. Frameworks that look identical in a demo diverge quickly under real CI noise, real data and real deploys.

The AI wrinkle

This month also brought OpenAI’s o1 reasoning models (announced 12 September). They’re the latest reminder that AI-assisted test authoring is improving fast. That has a framework implication people underrate. The framework with the most public examples is the one large language models write best. Copilot-style assistants produce more idiomatic Playwright and Cypress code than code for niche frameworks, simply because there is more of it to learn from. As AI test generation spreads, the network effects behind the download chart are likely to compound.

There’s also a counter-trend. Natural-language and “codeless” platforms such as mabl, testRigor, Testim and Functionize abstract the framework away entirely. For teams without strong automation engineers that can be the right trade, at the cost of portability. We look at that whole category in the November essay on AI testing agents.

Cross-browser coverage and cloud grids

Whatever you choose, real cross-browser and cross-device coverage usually means a cloud grid. BrowserStack, LambdaTest and Sauce Labs all support Playwright, Cypress and Selenium. Their support for BiDi sessions is still maturing, so check your vendor’s documentation before you rely on BiDi-only features in remote sessions. WebdriverIO’s own docs note that BiDi support depends on the remote environment.

Pros and cons in one place

Playwright

  • Pros: fast, free parallelism, multi-language, excellent debugging, strong momentum.
  • Cons: WebKit builds aren’t Safari; component testing still experimental; its own protocol layer rather than a W3C standard.

Cypress

  • Pros: best-in-class interactive DX, mature component testing, strong commercial cloud.
  • Cons: JS/TS only; architectural limits on multi-tab and cross-origin flows; parallelism mostly via a paid cloud.

Selenium 4

  • Pros: W3C standard, every major language, every major browser, huge ecosystem, BiDi momentum.
  • Cons: more assembly required for waits, reporting and debugging; perceived as “legacy”, which affects hiring.

What this means for your team

If you’re starting a new web project: default to Playwright unless you have a clear reason not to, such as a front-end team that loves Cypress for component tests or a Java/C# organisation standardised on Selenium. Write a one-page decision record either way.

If you’re on Cypress: there’s no urgency to move. Invest in test design and Cypress Cloud’s flake tooling. Revisit only if multi-tab, multi-origin or cost limits are actually hurting you.

If you’re on Selenium: upgrade to the latest 4.x and start experimenting with BiDi features in your language binding: console logs, network events and script evaluation. The standard is catching up with what Playwright offered, and you may not need to migrate at all.

For everyone:

  • Measure flake rate and time to diagnose per framework, not just runtime.
  • Keep test design framework-agnostic: page objects or screenplay patterns, stable data-testid or role-based locators, and isolated test data.
  • Watch the BiDi support matrix of your cloud grid vendor.

The framework wars are less interesting than they look. Protocols are converging, AI assistants are lowering authoring costs, and the remaining differences are about ergonomics and ecosystem. Pick deliberately, measure honestly and spend the saved energy on test design. That’s where reliability actually comes from.

Further reading on CheckOps