AWS Outage Lessons: Resilience Testing and Chaos Engineering

Two of the three largest cloud providers had major, widely felt outages within ten days of each other in late October 2025. Neither was caused by an attacker or a hardware failure. Both came down to software: automation behaving correctly in isolation and incorrectly together, and a configuration that passed its checks and then crashed the system that consumed it. In the same window OWASP published a draft Top 10 that, for the first time, names the mishandling of exceptional conditions as a top web application risk.

This essay covers those events and the other testing news from mid-October to mid-November. The deeper theme is resilience testing: how QA teams can test what their systems do when dependencies fail, and why “the cloud provider was down” is a scenario to test, not an excuse.

What happened

AWS us-east-1, 19–20 October

On 19–20 October, Amazon DynamoDB in us-east-1 suffered a disruption that spread to many AWS services and a long list of consumer apps. According to AWS’s post-event summary, the cause was a latent race condition in DynamoDB’s automated DNS management. One DNS “Enactor” was running slowly and applied an older DNS plan just after a faster Enactor had applied a newer plan and begun cleaning up old plans. The clean-up then deleted the plan that had just become active. That left the regional DynamoDB endpoint with an empty DNS record, an inconsistent state the automation couldn’t recover from by itself. Engineers had to intervene manually. AWS disabled the affected automation worldwide while it put fixes in place.

The dependency chain made it worse. Services that depend on DynamoDB, including parts of EC2’s instance-launch machinery, were affected in turn, and recovery for some of them took many hours after DNS was restored.

Azure Front Door, 29 October

On 29 October Azure Front Door had a global outage. Microsoft’s post-incident review describes a sequence of valid customer configuration changes, processed across two different control-plane build versions, that produced incompatible configuration metadata. That metadata passed staged health checks, propagated globally and was even saved as the “last known good” snapshot. Then an asynchronous process in the data plane hit a latent defect and crashed edge servers. Microsoft later published a lessons-learned post describing changes to validation and rollout.

The common thread

Put these next to the CrowdStrike incident from July 2024 and a pattern is clear:

Incident	Trigger	Why checks missed it
CrowdStrike, Jul 2024	Content update with a field-count mismatch	Validator and interpreter disagreed; no staged rollout
AWS us-east-1, Oct 2025	Race between two automation workers	Each worker behaved correctly; the interleaving didn’t
Azure Front Door, Oct 2025	Incompatible config across build versions	Health checks passed; the defect was in an asynchronous consumer

In each case the faulty component wasn’t the one that crashed. The defect lived in the interaction: between producer and consumer, between two workers, between two versions. Single-component tests, however good, didn’t exercise those interactions.

Resilience testing, defined

Resilience testing asks one question: when something we depend on fails or misbehaves, does our system degrade in the way we intended? It’s distinct from load testing, which asks how the system behaves under demand, and from functional testing, which asks whether it does the right thing when everything works.

A practical resilience test has four parts:

1. A dependency. A database, a cache, a third-party API, DNS, a cloud region, a feature-flag service.
2. A failure mode. Unavailable, slow, erroring, returning malformed data, or partially available.
3. An expected behaviour. Retry with backoff, fall back to cache, show a degraded UI, fail closed, queue for later.
4. An observable check. A user-visible state, a metric, a log line or an alert that proves the expected behaviour happened.

The third part is where most teams discover gaps. If nobody can say what the checkout page should do when the payment provider takes 30 seconds to respond, there’s no test to write yet. There’s a product decision to make first.

Exceptional conditions are now an OWASP Top 10 risk

In early November OWASP published the release candidate of the OWASP Top 10:2025. Two changes stand out for testers:

• A03:2025 Software Supply Chain Failures broadens the old “vulnerable and outdated components” category to cover dependencies, build systems and distribution infrastructure. That reflects the year we described in September.
• A10:2025 Mishandling of Exceptional Conditions is new. It covers improper error handling, logical errors, failing open and abnormal system states.

A10 is resilience testing viewed through a security lens. A system that “fails open”, granting access when the authorisation service times out, is both a resilience bug and a security vulnerability. The test is the same: inject the failure, check the behaviour.

A resilience test toolkit

Level 1: application-level fault injection in tests

The cheapest resilience tests run inside your existing test framework. With Playwright you can make a dependency fail for one test and assert on the degraded UI:

import { test, expect } from '@playwright/test';

test('checkout shows a retry message when payments API is down', async ({ page }) => {
  await page.route('**/api/payments/**', route => route.fulfill({ status: 503, body: 'unavailable' }));
  await page.goto('/checkout');
  await page.getByRole('button', { name: 'Pay now' }).click();
  await expect(page.getByRole('alert')).toContainText('try again');
  await expect(page.getByRole('button', { name: 'Pay now' })).toBeEnabled(); // no stuck spinner
});

test('recommendations panel degrades gracefully when slow', async ({ page }) => {
  await page.route('**/api/recommendations', async route => {
    await new Promise(r => setTimeout(r, 8_000));
    await route.continue();
  });
  await page.goto('/product/sku-123');
  // core content must not wait for the optional panel
  await expect(page.getByRole('heading', { level: 1 })).toBeVisible({ timeout: 3_000 });
  await expect(page.getByRole('button', { name: 'Add to basket' })).toBeEnabled();
});

For API and service tests, WireMock and Hoverfly can simulate faults such as delays, connection resets and malformed responses for downstream dependencies.

Level 2: network-level faults in integration environments

Toxiproxy sits between your service and a dependency and injects latency, bandwidth limits or connection drops on demand. It’s ideal for integration tests and works well with Testcontainers:

# proxy Redis through Toxiproxy, then add 1s latency
toxiproxy-cli create -l localhost:26379 -u localhost:6379 redis
toxiproxy-cli toxic add -t latency -a latency=1000 redis
# run the integration suite against localhost:26379, then remove the toxic
toxiproxy-cli toxic remove -n latency_downstream redis

Level 3: platform chaos experiments

In Kubernetes environments, tools such as Chaos Mesh and LitmusChaos define experiments as resources. Cloud providers offer managed options, such as AWS Fault Injection Service. A Chaos Mesh experiment that delays traffic to one service in staging:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payments-latency
  namespace: staging
spec:
  action: delay
  mode: all
  selector:
    namespaces: [staging]
    labelSelectors:
      app: payments
  delay:
    latency: '800ms'
    jitter: '200ms'
  duration: '10m'

Run a load test with k6 or Gatling during the experiment and assert on error rate and latency thresholds. That turns a chaos experiment into a pass/fail resilience test rather than a show.

Level 4: game days

A game day is a scheduled, supervised exercise. The team simulates a realistic failure, such as “our primary region’s database endpoint stops resolving”, and observes detection, communication and recovery. The output is a list of gaps: missing alerts, unclear runbooks, untested failovers. Game days test people and processes as well as software, which is exactly what the October outages stressed.

Retries, timeouts and the thundering herd

One resilience behaviour deserves special attention because it so often makes outages worse: retries. When a dependency fails, clients retry. When thousands of clients retry at once, without backoff or jitter, the recovering dependency gets hit by a wave of traffic larger than normal peak. Recovery from large outages often takes much longer than fixing the root cause for exactly this reason. Backlogs, retry storms and cold caches all have to drain.

Testers can check retry behaviour directly:

• Backoff exists. Fail a dependency and record the timestamps of retries. They should spread out over time, not arrive at a fixed interval.
• Jitter exists. Run several clients and check that their retries don’t line up.
• Retries are bounded. There’s a maximum attempt count or total time, after which the client fails cleanly with a clear message.
• Operations are idempotent. Retrying a payment or order creation mustn’t create duplicates. Use idempotency keys and test them.
• Timeouts are set everywhere. Every outbound call has an explicit timeout shorter than the caller’s timeout, so failures propagate quickly instead of piling up.

A simple service-level test using any HTTP mocking tool:

it('retries payments with exponential backoff and stops after 4 attempts', async () => {
  const calls: number[] = [];
  mockPayments.onPost('/charge').reply(() => { calls.push(Date.now()); return [503]; });

  await expect(chargeWithRetry(order)).rejects.toThrow('payment unavailable');

  expect(calls).toHaveLength(4);
  const gaps = calls.slice(1).map((t, i) => t - calls[i]);
  expect(gaps[1]).toBeGreaterThan(gaps[0]);   // growing delay
  expect(gaps[2]).toBeGreaterThan(gaps[1]);
});

Pair it with an idempotency test that sends the same request twice with the same key and asserts one charge.

Measuring resilience with SLOs

Resilience tests need pass/fail criteria, and service level objectives provide them. If your checkout SLO is “99.9% of checkout requests succeed within 2 seconds over 30 days”, a chaos experiment that delays the payments service is a direct question: does the SLO hold during the experiment, and how much error budget would a real event of this length consume?

Experiment	SLO signal to assert on	Example pass criterion
Payments API +800 ms latency	Checkout p95 latency, success rate	p95 under 2 s, success above 99.5% during experiment
Recommendations API down	Product page availability	100% of product pages render core content
Cache cluster restart	Database load, page latency	DB CPU stays under 70%; p95 under 1.5 s within 5 min
Auth provider timeouts	Protected endpoint behaviour	0 unauthorised successes (fail closed)

Writing the criteria down before the experiment is the discipline that separates chaos engineering from chaos.

The scenarios October suggests

Scenario	What to test	Expected behaviour (example)
Regional managed-database endpoint doesn’t resolve	DNS failure for a primary dependency	Fail over or serve read-only mode; alert within minutes
CDN or front door unavailable	Edge layer down	Status page reachable from separate infrastructure; clear messaging
Auth provider times out	Slow or unavailable identity service	Fail closed for protected actions; existing sessions honoured per policy
Feature-flag service unreachable	Config dependency down	SDK falls back to safe cached defaults
Third-party API returns malformed JSON	Bad data from dependency	Validate, log, degrade; never crash the page
Two workers race on shared state	Concurrency	Idempotent operations; no destructive clean-up of active state
Config valid in v1, incompatible in v2	Mixed versions during rollout	Consumers reject unknown config safely; rollout halts

The last two rows map directly onto the AWS and Azure root causes. Concurrency and mixed-version testing are hard, but property-based tests, deterministic simulation and contract tests between config producers and consumers all help.

Testing the feature-flag fallback

Feature-flag services deserve their own test, because nearly every modern app depends on one at startup. Most SDKs, including LaunchDarkly’s, take a default value in each evaluation call. That default is what you get when the service is unreachable. A unit-level test of your wrapper makes the fallback explicit:

import { describe, it, expect, vi } from 'vitest';
import { isNewCheckoutEnabled } from '../flags';

describe('flag fallbacks', () => {
  it('defaults to the safe value when the flag service is unreachable', async () => {
    const client = { variation: vi.fn().mockRejectedValue(new Error('ECONNREFUSED')) };
    await expect(isNewCheckoutEnabled(client as any, { key: 'u1' })).resolves.toBe(false);
  });
});

Then decide, as a product question, what “safe” means for each flag, and write it down next to the flag.

Also in the window

Cypress cy.prompt goes public

In October Cypress made cy.prompt available to everyone as an experimental feature, starting with Cypress 15.4. It converts natural-language steps into Cypress commands, can self-heal when selectors change, and shows the generated commands in the Command Log. You can keep the prompt in the test or eject the generated code:

// cypress.config.js: enable with experimentalPromptCommand: true (requires Cypress Cloud)
it('logs in', () => {
  cy.prompt([
    'visit the login page',
    'type "qa@example.test" into the email field',
    'type the password from env TEST_PASSWORD',
    'click the sign-in button',
  ]);
  cy.contains('Welcome back').should('be.visible');
});

Note the deterministic assertion after the prompt. That’s the pattern we’d recommend: natural-language steps for setup and navigation, explicit assertions for what matters. Cypress now joins Playwright’s Test Agents in shipping AI authoring and healing in the framework itself, and the Playwright vs Cypress comparison has acquired an AI dimension.

World Quality Report 2025–26

The 17th World Quality Report, published on 13 November by Capgemini, Sogeti and OpenText, found that 89% of organisations are piloting or deploying generative AI in quality engineering, but only 15% have achieved enterprise-wide deployment. Average reported productivity gains were 19%, and about a third of organisations saw minimal benefit. Compared with the 68% “using or planning” figure in last year’s report, which we covered in November 2024, adoption has broadened considerably while scaling remains hard.

Making resilience testing routine

Resilience testing fails when it’s a one-off project. To make it routine:

• Write expected degraded behaviour into acceptance criteria. “When the recommendations API is slow, the product page renders core content within 2 seconds” is testable.
• Keep a dependency register. List each dependency, its failure modes, the expected behaviour and the test that covers it. Review it quarterly.
• Run level-1 tests in every pull request. They’re cheap and fast.
• Run level-2 and level-3 tests on a schedule. Weekly in staging, with results on the quality dashboard.
• Hold a game day every quarter. Rotate the scenario. Use real incidents from your industry, such as October’s, as prompts.
• Track resilience findings like bugs. With severity, owner and due date.

Pros and cons of investing in resilience testing

Pros

• It targets the failure class behind the largest outages of the past two years.
• Level-1 tests are cheap and use frameworks you already have.
• It overlaps with security testing, through A10:2025 and failing closed.
• It improves incident response through game days and runbooks.

Cons

• Expected degraded behaviour often needs product decisions first.
• Platform chaos needs mature environments and careful safety controls.
• Concurrency and mixed-version bugs remain hard to reproduce.
• It can feel like work for scenarios that “never happen” until they do.

What this means for your team

Pick your top five dependencies and write one failure test for each this month. Use route interception or service virtualisation. No new infrastructure is needed.

Add “what happens when X is down?” to your test design checklist. Make degraded behaviour an acceptance criterion, not an afterthought.

Test fail-closed behaviour for security-relevant dependencies. Auth, authorisation and payment providers first. A10:2025 makes this a security requirement as well as a reliability one.

Contract-test configuration. Treat config producers and consumers like API producers and consumers. Validate config with the same parser that consumes it, across the versions that will coexist during rollout.

Run a game day based on the AWS outage. Simulate a primary database endpoint that stops resolving and measure detection time, communication and recovery.

Keep AI authoring honest. Whether you adopt cy.prompt or Playwright Test Agents, keep assertions explicit and reviewable.

October showed that the most resilient-looking platforms in the world still fail in ways their own checks didn’t anticipate. You can’t prevent your providers’ outages. You can test what your product does during them. For tools to help, see Best performance testing tools 2025 and Best security testing tools 2025.

Further reading on CheckOps

• CrowdStrike outage lessons for release quality testing
• Shift-left meets shift-right: observability-driven testing
• npm supply chain attacks: security testing for QA teams