Shift-Left Meets Shift-Right: Sonos and Observability

“Shift left” has been the dominant QA slogan for a decade: test earlier, test closer to the code, catch defects before they get expensive. “Shift right”, testing in and learning from production, has been its quieter sibling, usually owned by SRE rather than QA. The past four weeks offered a textbook case of what happens when neither is done well. They also brought a few tooling moves that make doing both cheaper.

This essay covers mid-September to mid-October 2024. The anchor story is Sonos. The deeper theme is that shift-left and shift-right aren’t competing philosophies. They’re the two halves of one feedback loop, and observability tooling such as OpenTelemetry is what joins them.

Sonos: a release-quality case study in public

Sonos shipped a rebuilt mobile app in May 2024. It removed features customers relied on, introduced bugs in core flows, and triggered months of public anger. In August, the company said fixing it would cost $20–30 million. We touched on that in our CrowdStrike release-quality essay.

On 23 September, Bloomberg published an account of how the launch happened. Follow-up coverage by The Verge and Ars Technica summarised it. Employees had warned that the app wasn’t ready. There was a list of bugs considered essential to fix before launch, but it wasn’t comprehensive. The company went ahead anyway.

On 1 October Sonos announced seven quality and customer-experience commitments, and its CEO told Reuters the executive team would forgo bonuses unless quality improved. The commitments are worth reading as a QA document:

1. More stringent pre-release quality benchmarks.
2. A broader beta-testing programme with customers.
3. Gradual introduction of major app changes, with experimental features as opt-in.
4. An internal quality ombudsperson.
5. Extended warranties on some speakers.
6. Regular app updates on a predictable cadence.
7. A customer advisory board.

Look at that list through a testing lens. Items 1 and 4 are shift-left: stronger gates and a person whose job is to say “not yet”. Items 2, 3 and 6 are shift-right: real users, staged exposure and small, frequent changes that are easier to observe and roll back. The company didn’t pick one. It needed both, and the absence of both caused the damage.

Shift-left: what it actually means in 2024

Shift-left gets reduced to “developers write more unit tests”. That’s part of it, but the useful definition is broader: move every kind of feedback as close as possible to the moment a decision is made.

Decision point	Shift-left practice	Example tools
Requirement written	Testability review, example mapping	Three amigos, BDD scenarios
Code written	Unit and component tests, static analysis	Vitest, Jest, Semgrep, SonarQube
API changed	Contract tests	Pact, PactFlow, Microcks
Dependency added	Software composition analysis	Snyk, OWASP Dependency-Check
Merge requested	Targeted E2E and security scans	Playwright, ZAP

Contract testing: the most underused shift-left technique

If there’s one shift-left technique most teams under-invest in, it’s consumer-driven contract testing. End-to-end tests across microservices are slow, flaky and expensive. Contract tests check the same integration point, “does the provider still give the consumer what it needs?”, in milliseconds and without deploying anything.

A minimal Pact consumer test in JavaScript:

const { PactV3, MatchersV3 } = require('@pact-foundation/pact');
const { like, eachLike } = MatchersV3;

const provider = new PactV3({ consumer: 'mobile-app', provider: 'playback-api' });

it('lists rooms for a household', () => {
  provider
    .given('household 42 has two rooms')
    .uponReceiving('a request for rooms')
    .withRequest({ method: 'GET', path: '/households/42/rooms' })
    .willRespondWith({
      status: 200,
      body: eachLike({ id: like('r1'), name: like('Kitchen'), volume: like(30) }),
    });

  return provider.executeTest(async (mock) => {
    const rooms = await client(mock.url).listRooms(42);
    expect(rooms[0].name).toBeDefined();
  });
});

The provider team then verifies that contract in its own pipeline. If a provider change would break a consumer, it fails before merge. For a product like Sonos, where a mobile app, a desktop controller, a cloud API and speaker firmware all talk to each other, contract tests are among the few practical ways to catch integration regressions early.

Security shifts left too: ZAP joins Checkmarx

On 24 September the ZAP project announced that its core team had joined Checkmarx, and the project is now branded “ZAP by Checkmarx”. ZAP stays open source under Apache-2.0. The move gives the most widely used open-source DAST tool full-time funded maintainers inside a large application-security vendor.

ZAP’s baseline scan is one of the easiest shift-left security wins available, and it fits on a single CI step:

- name: ZAP baseline scan
  uses: zaproxy/action-baseline@v0.12.0
  with:
    target: ${{ env.PREVIEW_URL }}
    rules_file_name: .zap/rules.tsv
    fail_action: true

We cover the wider landscape in our security testing tools 2025 roundup.

AI code quality gets funded

On 30 September Qodo (formerly CodiumAI) raised a $40M Series A for its quality-focused AI coding platform, which includes test generation and code review. Whatever you think of AI test generation, this is shift-left in its purest form: tests suggested while the code is being written. We look at how well that works in practice in our essay on AI coding agents writing tests.

Shift-right: testing in production without being reckless

Shift-right makes some QA engineers nervous because it sounds like “let users find the bugs”. Done properly, it’s the opposite. It’s a set of techniques for getting quality signals that no pre-production environment can produce:

• Progressive delivery. Canary releases, deployment rings and percentage rollouts behind feature flags such as LaunchDarkly.
• Synthetic monitoring. Scripted journeys running continuously against production, often with the same Playwright code as your E2E suite.
• Real-user monitoring and error tracking. Front-end performance, crash rates and JavaScript errors from real sessions with tools like Sentry, Datadog and New Relic.
• Chaos and resilience experiments. Deliberately injecting failure to verify that fallbacks work. We return to this in our AWS outage essay.
• Beta and opt-in programmes. Exactly what Sonos committed to.

The connective tissue is observability. If you can’t see what the system is doing in production, shift-right is just hoping.

OpenTelemetry: the layer that joins left and right

OpenTelemetry (OTel) is the CNCF’s vendor-neutral standard for traces, metrics and logs. For QA its value is simple. The same instrumentation that powers production dashboards can power test assertions.

Trace-based testing

A classic E2E test asserts on what the user sees. A trace-based test can also assert on what happened inside the system during that request: which services were called, how long each span took, whether a retry happened, whether a cache was hit.

Here’s a sketch of the idea in a Playwright test. The test propagates a W3C traceparent header and queries the tracing backend afterwards:

import { randomBytes } from 'crypto';

test('checkout does not call inventory twice', async ({ page, request }) => {
  const traceId = randomBytes(16).toString('hex');
  await page.setExtraHTTPHeaders({ traceparent: `00-${traceId}-${randomBytes(8).toString('hex')}-01` });

  await page.goto('/checkout');
  await page.getByRole('button', { name: 'Place order' }).click();
  await expect(page.getByRole('status')).toHaveText(/confirmed/i);

  const spans = await (await request.get(`${TRACES_API}/traces/${traceId}`)).json();
  const inventoryCalls = spans.filter((s: any) => s.name === 'GET /inventory');
  expect(inventoryCalls.length).toBe(1);
});

Dedicated tools take this further. Tracetest and similar products let you write assertions directly against spans. Grafana Tempo, Jaeger and commercial APMs expose trace APIs you can query from tests. The point isn’t the tool. The point is that the test and the production dashboard look at the same data.

Why this matters for the left-right loop

When tests and production share instrumentation, three good things happen:

1. Production incidents become test cases faster. A trace from a failing production request shows exactly which path to reproduce.
2. Test failures become easier to diagnose. A red E2E run with a trace attached points straight at the slow or failing service.
3. Coverage gaps become visible. Compare the span names your test suite produces with the ones production produces. Paths that only appear in production are untested paths.

That third point is underrated. Production telemetry is the most honest coverage report you’ll ever get.

Synthetic monitoring: reuse your E2E suite in production

One of the cheapest shift-right investments is to stop maintaining two separate sets of scripted journeys. Many teams have an E2E suite owned by QA and a synthetic monitoring suite owned by SRE, written in different tools, drifting apart and testing slightly different things. A better pattern is to tag a small subset of E2E tests as production-safe and run them on a schedule against production.

// playwright.config.ts
export default defineConfig({
  projects: [
    { name: 'e2e', testMatch: /.*\.spec\.ts/, use: { baseURL: process.env.STAGING_URL } },
    {
      name: 'synthetics',
      grep: /@synthetic/,
      use: { baseURL: process.env.PROD_URL },
      retries: 1,
    },
  ],
});

test('search returns results @synthetic', async ({ page }) => {
  await page.goto('/');
  await page.getByRole('searchbox').fill('speaker');
  await expect(page.getByRole('listitem').first()).toBeVisible();
});

The rules for production-safe tests are strict. They must be read-only or use dedicated test accounts. They must never create real orders or send real messages. They must tolerate production data changing under them. Within those constraints you get continuous, user-level verification for the price of a scheduled CI job, and your E2E suite gets a reality check every few minutes.

Release gates as code

The first Sonos commitment, “more stringent quality benchmarks”, is only as strong as its enforcement. Benchmarks that live in a slide deck get waived under deadline pressure. Benchmarks in the pipeline are harder to ignore. Here’s an illustrative release gate, written as a policy file a pipeline step evaluates before promoting a build:

# release-gates.yaml
gates:
  pre_release:
    critical_journeys_pass_rate: ">= 100%"
    open_p1_defects: "== 0"
    contract_verification: "all consumers green"
    zap_high_alerts: "== 0"
  canary:
    duration: 2h
    crash_free_sessions: ">= 99.5%"
    error_rate_delta_vs_baseline: "<= +0.2pp"
    p95_latency_delta_vs_baseline: "<= +10%"
  rollback_triggers:
    crash_free_sessions: "< 99.0%"
    checkout_success_rate_delta: "<= -1pp"

Two details matter more than the specific numbers. First, canary metrics are compared against a baseline of the current production version, not against absolute thresholds, because absolute numbers drift with traffic. Second, there’s an explicit owner with authority to block. That’s the role Sonos describes as a quality ombudsperson. Without one, every gate becomes negotiable.

Beta programmes are a test technique, so design them like one

Sonos promised a broader customer beta. Beta programmes often disappoint because they’re treated as marketing rather than as a test design problem. A useful beta is designed around questions:

Question	Beta design choice
Do existing power users lose anything?	Recruit long-tenure users with complex setups, not just enthusiasts
Does it work across the hardware matrix?	Stratify by device model, OS version and network type
Are we fixing the right bugs first?	In-app reporting with logs and traces attached automatically
Is the new version better?	Compare task-success and crash metrics between beta and stable cohorts

The last row is where observability matters again. Without comparable telemetry between beta and stable, a beta yields anecdotes. With it, you get evidence.

Playwright 1.48: testing the real-time paths

Playwright 1.48 shipped on 8 October with page.routeWebSocket(), which lets tests intercept, mock or modify WebSocket traffic. Real-time paths such as live dashboards, chat, collaborative editing and, yes, controlling a speaker from your phone have historically been hard to test deterministically. Mocking the socket makes the failure modes testable:

await page.routeWebSocket('wss://api.example.test/live', ws => {
  ws.onMessage(msg => {
    if (msg === 'subscribe:room-1') ws.send(JSON.stringify({ room: 'room-1', state: 'playing' }));
  });
});

It’s a small API, but it fills a real gap in shift-left coverage for event-driven UIs.

Metrics that tie both directions together

If shift-left and shift-right are one loop, they need shared metrics. The four DORA software delivery metrics are the closest thing the industry has to a common language:

• Deployment frequency. How often you ship to production.
• Lead time for changes. How long it takes a commit to reach production.
• Change failure rate. The share of deployments that cause a failure needing remediation.
• Time to restore service. How quickly you recover when they do.

Shift-left practices mainly improve the first two: fast, trustworthy pipelines let you ship often with short lead times. Shift-right practices mainly improve the last two: progressive delivery limits the blast radius of bad changes, and observability shortens recovery. A team that only invests left often ends up shipping quickly into incidents it detects slowly. A team that only invests right ends up with excellent dashboards showing a steady stream of avoidable failures.

QA teams can add two metrics of their own to the DORA four:

1. Escaped defect rate by origin. For every production defect, record which layer should have caught it: unit, contract, E2E, security scan, canary. Over a quarter this shows exactly where your loop is weakest.
2. Time from production signal to regression test. When an incident or customer bug is found, how long until a test exists that would catch it next time? Mature teams measure this in days. Many teams never close the loop at all.

Both metrics are cheap to collect with a field in your incident template and a label in your issue tracker. They turn “we should test more” into a specific, prioritised investment list.

The community mood: what’s broken in test automation

Software Testing Weekly’s mid-October issue featured a widely shared discussion of what’s broken in test automation. The themes will be familiar: flaky suites, slow pipelines, tests no one trusts, and automation measured by test count rather than risk reduction. Read next to the Sonos story, the diagnosis is uncomfortable. Organisations can have plenty of automation and still ship a broken release if the automation doesn’t answer the questions leadership is actually asking:

• Are the flows customers rely on still working?
• Is this release better than the last one on the metrics we care about?
• If something goes wrong, how fast will we know and how fast can we undo it?

Shift-left answers the first question before release. Shift-right answers all three after release. You need both to answer them with confidence.

Pros and cons of investing in each direction

Shift-left: pros

• Cheapest defects to fix.
• Fast feedback for developers.
• Contract and static checks scale well across many teams.

Shift-left: cons

• Can’t reproduce production scale, data or device diversity.
• Risk of false confidence: “all green” isn’t “works for users”.
• Needs developer buy-in and good test design to avoid brittle suites.

Shift-right: pros

• The only source of truth about real user experience.
• Catches emergent issues: performance at scale, rare devices, data drift.
• Turns rollouts into experiments with automatic safety nets.

Shift-right: cons

• Requires investment in observability, flags and rollout tooling.
• Some users are exposed to defects, even if only a small percentage.
• Privacy and compliance constraints on what you can collect.

What this means for your team

Map your loop. For your three most important user journeys, write down the shift-left checks (unit, contract, E2E, security) and the shift-right signals (canary metrics, synthetics, RUM, error rates). Wherever one side is empty, you’ve found your next investment.

Start one contract test. Pick the integration that has broken most often in the past year and add a Pact contract between consumer and provider. Measure how many regressions it catches over a quarter.

Add a ZAP baseline scan to pull requests. It takes an afternoon and produces a steady stream of real findings, mostly headers and cookie flags, that are cheap to fix early.

Instrument once, assert twice. If you’re adopting OpenTelemetry for production, ask for trace context propagation in test environments too. Then write at least one trace-based assertion for a critical path.

Make release gates explicit. Sonos’s “more stringent quality benchmarks” only work if they’re written down and enforced. Define the error-rate, crash-rate and journey-success thresholds that block a release, and the ones that trigger automatic rollback.

Ship smaller. Every technique in this essay gets easier when releases are smaller and more frequent. A “big bang” rewrite is the hardest thing to test from either direction.

The Sonos commitments will be judged over the next year by customers, not by blog posts. The principle behind them applies everywhere: quality is a loop, not a gate. Shift left to make the loop fast. Shift right to make it honest.

Further reading on CheckOps

• CrowdStrike outage lessons for release quality testing
• Playwright vs Cypress vs Selenium: the BiDi shift
• AI testing agents arrive: computer use and DORA 2024