npm Supply Chain Attacks: Security Testing for QA Teams

Test automation runs on JavaScript more than on any other ecosystem. Playwright, Cypress, WebdriverIO, Puppeteer, Appium and most of their plugins ship through npm, and they run on CI machines that hold deploy keys, cloud credentials and package-publishing tokens. Between mid-August and mid-September 2025 that combination stopped being a theoretical risk. Two of the most serious npm compromises to date hit packages found in a huge number of node_modules folders, including the ones under your test suites.

This essay covers those incidents and the testing news that came alongside them. The deeper theme is supply-chain security testing as a QA responsibility: your test infrastructure is part of your attack surface, and the controls that protect it are testable.

What happened

Nx “s1ngularity”, 26 August

On 26 August, attackers published malicious versions of Nx, the widely used build system for JavaScript monorepos, along with several of its plugins. Affected versions included nx 20.9.0–20.12.0 and 21.5.0–21.8.0. A postinstall script scanned developer machines and CI runners for credentials, SSH keys, .env files and crypto wallets, then uploaded what it found to public GitHub repositories named s1ngularity-repository.

One detail made this attack a first. The payload tried to use locally installed AI coding CLIs, including Claude Code, Gemini CLI and Amazon Q, with permission-bypassing flags to help search the file system for secrets. The AI agents we’ve spent this year adopting were treated as a post-exploitation tool. Nx later published a detailed post-mortem. The root cause was a vulnerable GitHub Actions workflow that allowed theft of the npm publishing token.

chalk, debug and 16 more, 8 September

On 8 September a maintainer of some of the most depended-upon packages in the ecosystem was phished through a fake npm support domain. Attackers then published malicious versions of 18 packages, including chalk 5.6.1 and debug 4.4.2, with roughly 2.6 billion combined weekly downloads. The payload targeted browsers: it hooked web APIs to silently redirect crypto-wallet transactions.

The community detected the compromise within hours, and the malicious versions were removed quickly. A later analysis estimated that the attackers made almost nothing. That was luck and fast detection, not design. The same access could have shipped a credential stealer into any CI pipeline that installed fresh dependency versions that afternoon.

Why test infrastructure is a prime target

Test pipelines combine several risk factors:

Risk factor	Why it matters in test pipelines
Huge dependency trees	A typical E2E project pulls in hundreds of transitive packages
Frequent fresh installs	CI runs npm install many times a day, often without a lockfile check
Privileged environments	Runners hold cloud credentials, test-account passwords and sometimes publish tokens
Install scripts	postinstall hooks execute arbitrary code at install time
Browser downloads	Frameworks download browser binaries and drivers during setup
AI agents on dev machines	Agents with broad file access can be abused, as Nx showed
Low ownership	“It’s just the test repo” often means weaker review and fewer controls

The last row is the cultural problem. Test repositories frequently get looser branch protection, fewer reviewers and older dependencies than production code, while running with comparable or greater privileges.

A supply-chain security testing playbook

These are controls you can implement and test in a test-automation codebase. Most take minutes.

1. Lock and verify dependencies

Use lockfiles everywhere, and install with the command that refuses to change them:

# fails if package-lock.json and package.json disagree; never rewrites the lockfile
npm ci
# verify registry signatures and provenance attestations for installed packages
npm audit signatures

Add a CI check that fails if a pull request modifies package.json without a corresponding lockfile change, or the reverse.

2. Delay brand-new versions

Most malicious versions are detected and removed within hours or days. A minimum release age gives the ecosystem time to catch them before you install. pnpm 10.16, released on 12 September, added a minimumReleaseAge setting for exactly this:

# pnpm-workspace.yaml
minimumReleaseAge: 4320   # minutes: only install versions published at least 3 days ago

If you use npm or Yarn, you can get a similar effect from Renovate’s or Dependabot’s scheduling and cooldown settings, combined with a strict lockfile.

3. Disable install scripts by default

# .npmrc
ignore-scripts=true

Then explicitly allow the few packages that genuinely need scripts. Browser installers are a common example, and you can usually run them as a separate, explicit step such as npx playwright install --with-deps. pnpm 10 already blocks dependency lifecycle scripts unless you allow them.

4. Scan dependencies continuously

Run a software composition analysis (SCA) tool on every pull request and on a nightly schedule, because new advisories appear against code you haven’t changed. Options include Snyk, OWASP Dependency-Check, Black Duck and GitHub’s own Dependabot alerts.

# .github/workflows/deps.yml (excerpt)
on:
  pull_request:
  schedule: [{ cron: '0 3 * * *' }]
jobs:
  sca:
    runs-on: ubuntu-latest
    permissions: { contents: read }
    steps:
      - uses: actions/checkout@v4
      - run: npm ci --ignore-scripts
      - run: npm audit --audit-level=high
      - run: npm audit signatures

5. Harden CI workflows

The Nx compromise started with a workflow weakness. Treat workflow files as security-critical code:

• Set permissions: explicitly, read-only by default, on every workflow.
• Avoid pull_request_target with checkout of untrusted code.
• Pin third-party actions to a full commit SHA rather than a mutable tag.
• Use OIDC and short-lived credentials instead of long-lived secrets.
• Publish packages with npm trusted publishing and provenance, so a stolen token isn’t enough.

permissions:
  contents: read
steps:
  - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0 pinned by SHA

A static analysis tool for workflows, or a simple script that fails on unpinned uses: lines, turns this policy into a test.

6. Separate secrets from test jobs

E2E tests rarely need production credentials. Give test jobs their own scoped test-environment secrets, never the deploy or publish tokens. If a malicious package runs during npm ci, its blast radius is then a test account, not your cloud.

7. Constrain AI agents on developer machines

After Nx, agent permissions are a supply-chain control. Don’t run coding agents with permission-bypass flags on machines that hold real credentials. Use the agent’s allow-list configuration to restrict shell commands and file paths. Keep secrets in a credential manager rather than plaintext .env files.

8. Generate and keep SBOMs

A software bill of materials for each build lets you answer “were we affected?” in minutes when the next advisory drops:

npx @cyclonedx/cyclonedx-npm --output-file sbom.json

Store the SBOM as a build artefact, and keep an incident-response script that greps your SBOMs for affected package versions.

What E2E tests can catch: browser-side payloads

The chalk and debug payload is worth a closer look, because it ran in the browser. Packages that end up in front-end bundles execute on your users’ devices, and that’s somewhere end-to-end tests already reach. Two cheap checks would raise an alarm when a bundle suddenly starts behaving differently.

Fail on unexpected network destinations. Most applications talk to a known, short list of origins. A Playwright fixture can record every request in E2E runs and fail the test if anything goes elsewhere:

// fixtures/network-allowlist.ts
import { test as base, expect } from '@playwright/test';

const ALLOWED = [/^https:\/\/(www\.)?example\.com/, /^https:\/\/api\.example\.com/, /^https:\/\/cdn\.example\.com/];

export const test = base.extend({
  page: async ({ page }, use) => {
    const unexpected: string[] = [];
    page.on('request', req => {
      const url = req.url();
      if (url.startsWith('data:') || url.startsWith('blob:')) return;
      if (!ALLOWED.some(rx => rx.test(url))) unexpected.push(url);
    });
    await use(page);
    expect(unexpected, 'requests to non-allow-listed origins').toEqual([]);
  },
});

Enforce and monitor a Content Security Policy. A strict connect-src and script-src policy in staging, with violations reported and E2E tests failing on any console CSP error, turns unexpected behaviour into a visible failure rather than a silent one.

Watch bundle diffs. A CI step that compares bundle size and the list of included packages against the previous build, and flags large unexplained changes, is crude but effective. Malicious code has to live somewhere, and new obfuscated blobs in a bundle are worth a human look.

None of these is a complete defence. They work as tripwires, and tripwires are what shorten the gap between compromise and detection.

Dependency hygiene for test projects

The cheapest supply-chain control is having fewer dependencies. Test projects collect packages over the years: reporters nobody reads, helper libraries replaced by built-in framework features, and three different assertion libraries. A quarterly clean-up is worth doing:

1. Find unused packages. Tools such as depcheck or knip list dependencies that no code imports.
2. Replace helpers with built-ins. Modern Playwright, Cypress and Vitest cover retries, fixtures, mocking and reporting natively. Many older helper packages are now redundant.
3. Consolidate frameworks. If you run two E2E frameworks, the migration cost may be justified by the halved dependency surface alone.
4. Prefer well-maintained packages. Check release activity, maintainer count and whether the package publishes with provenance before adding it.
5. Track the count. Put the number of direct and transitive dependencies on your quality dashboard. What gets measured tends to shrink.

npx knip --dependencies          # unused and unlisted dependencies
npm ls --all --parseable | wc -l  # rough transitive dependency count

A test project with 40 direct dependencies and 600 transitive ones isn’t unusual. Cutting that by a third reduces the attack surface, speeds up npm ci and shrinks the upgrade backlog, all at once.

Testing your supply-chain controls

Controls that aren’t tested drift. A few checks we’d add to a quarterly security test run:

Control	How to test it
Lockfile enforcement	Open a PR that changes package.json only; CI should fail
Install scripts disabled	Add a throwaway local package with a postinstall that writes a file; verify it doesn’t run
Minimum release age	Attempt to install a version published today; it should be refused
Workflow permissions	Script that fails on any workflow without an explicit permissions: block
Action pinning	Script that fails on uses: lines not pinned to a 40-character SHA
Secret scoping	Review which secrets each job can read; E2E jobs should have no deploy or publish tokens
Incident lookup	Time how long it takes to answer “do we use chalk 5.6.1 anywhere?”

That last row is a good drill. If the honest answer is “a day of grepping repositories”, invest in SBOMs and centralised dependency inventory.

Other testing news from the window

Appium 3

Appium 3.0 shipped on 18 August. It requires Node.js 20.19 or newer, removes deprecated endpoints and legacy JSON Wire Protocol parameters on shared W3C endpoints, and requires a driver scope prefix for insecure features, such as --allow-insecure=uiautomator2:adb_shell. For Appium users the migration guide is short, but old client code that still sends desiredCapabilities will break. Search your mobile test code for it before upgrading.

Playwright 1.55

Playwright 1.55 (August) added automatic toBeVisible() assertions to codegen, as an opt-in setting, and a testStepInfo.titlePath property. It also dropped support for Manifest V2 Chromium extensions, which matters if you test a browser extension. Codegen that writes assertions as well as actions is a small but meaningful improvement for teams that record a first draft and then refine it.

The Playwright vs Cypress conversation continues

Both frameworks kept shipping through the summer, and community discussion of Playwright vs Cypress stayed one of the most searched topics in test automation. From a supply-chain perspective there’s one practical point to add to that comparison. Every framework and plugin you add increases the dependency surface your CI installs. Teams that standardise on one primary framework, with a small, reviewed plugin set, reduce both maintenance cost and supply-chain risk. Our framework comparison covers the other trade-offs.

Questions to ask your testing vendors

Commercial testing platforms, cloud grids and AI testing tools also run code in your pipelines and often hold credentials to your environments. Add a few supply-chain questions to your next vendor review:

• How do you publish your CLI, SDK and plugins? Do the packages carry provenance attestations, and is publishing protected by trusted publishing or hardware-backed 2FA?
• What do your packages execute at install time?
• What credentials does your agent or tunnel need, and can they be scoped to a test environment only?
• How quickly did you assess exposure to the Nx and chalk/debug incidents, and how did you tell customers?
• Can you provide an SBOM for the components you ship to us?

A vendor that answers these clearly and quickly is likely to handle the next incident well. Vague answers tell you something too.

Incident response for a compromised dependency

When the next advisory lands, a test team should be able to run this checklist in under an hour:

1. Identify. Search SBOMs and lockfiles across all repositories, test repositories included, for the affected package versions.
2. Contain. Block the versions in your registry proxy or package manager configuration, and pause scheduled pipelines that would install fresh versions.
3. Assess. For each repository that installed an affected version, determine when, on which machines and runners, and which secrets were available there.
4. Rotate. Rotate every credential that was present in an affected environment. Don’t try to prove it wasn’t stolen; assume it was.
5. Clean. Clear dependency caches, including CI caches, which are easy to forget, and rebuild from a known-good lockfile.
6. Review. Record what made detection and response slow, and fix one thing before the next incident.

CI caches deserve special mention. Many pipelines cache node_modules or the package manager store between runs. A malicious version cached on the afternoon of an attack can be restored days later, after the registry has removed it. Your clean-up has to include cache invalidation.

Pros and cons of strict supply-chain controls

Pros

• They reduce exposure to the most common and fastest-growing attack class.
• Most controls are configuration, not new tools.
• They improve reproducibility and reduce flaky “works on my machine” installs.
• They make incident response fast and evidence-based.

Cons

• A minimum release age delays urgent security fixes unless you add an override process.
• Disabling install scripts breaks some packages until you allow them.
• SHA-pinned actions need automated update tooling to stay current.
• Extra checks add CI minutes and some friction for contributors.

What this means for your team

Bring test repositories up to production standards. Same branch protection, same review rules, same dependency policies. A test repo with deploy credentials is production infrastructure.

Add the five-minute controls this week. npm ci, npm audit signatures, explicit workflow permissions, ignore-scripts with an allow-list, and scoped secrets for test jobs.

Adopt a release-age delay. Three to seven days catches most malicious publishes. Document an override path for urgent fixes.

Make supply-chain checks testable. Use the control table above as a quarterly drill, and record how long the “are we affected?” lookup takes.

Include AI agents in the threat model. Agents on machines with secrets need allow-lists, not bypass flags. Nx proved attackers will use whatever tools are installed.

Own the incident drill. QA teams are good at runbooks and reproducible checks. Write the dependency-compromise runbook and rehearse it before you need it.

Security testing isn’t only scanning the application. In 2025 it also means testing the pipeline that builds and tests the application, because attackers have noticed that’s where the keys are. For a full tool comparison, see Best security testing tools 2025.

Further reading on CheckOps

• CrowdStrike outage lessons for release quality
• MCP and the OWASP LLM Top 10
• LLM testing and evals in the vibe-coding era