Essay
The QA Engineer Role in the AI Era: 2025 Year in Review
Cloudflare's outage, React2Shell, Shai-Hulud 2.0, the Agentic AI Foundation and OWASP's agentic Top 10 close out 2025. What the year means for QA engineers.
15 December 2025 · 11 min · Irfan Ahmad
At the start of 2025 the question in many QA teams was whether AI would change testing. By December the question was how fast, and who would own the parts that matter. Coding agents open pull requests at scale. Playwright and Cypress ship AI authoring and healing. The protocol connecting agents to tools has moved into a neutral foundation. And the year ended the way it went on: with an outage caused by a file that passed its checks, a critical vulnerability in one of the most widely used front-end frameworks, and another npm worm.
This final monthly essay covers the four weeks to mid-December, then steps back. The deeper theme is the QA engineer role in the AI era: what changed in 2025, what didn’t, and where testers add the most value going into 2026.
The last four weeks
Cloudflare, 18 November
On 18 November a large part of the web returned errors for several hours. Cloudflare’s post-mortem is a model of transparency, and a textbook testing case. A permissions change in a ClickHouse database cluster caused a query to return duplicate rows. That query generated the “feature file” used by Cloudflare’s Bot Management system. The file roughly doubled, to more than 200 features. The proxy software had a preallocated limit of 200 features, and exceeding it caused an unhandled error (a Rust panic) and 5xx responses across the network. Engineers stopped propagation of the bad file, restored a known-good version, and resolved the main impact the same day.
Put it next to CrowdStrike in 2024 and AWS and Azure in October:
- A generated configuration artefact was treated as trusted input.
- A hard limit in the consumer was exceeded, with no graceful handling.
- The change that triggered it, a database permission, was far from the code that failed.
- Fast global propagation turned a local defect into a global outage.
The tests that would have helped are familiar by now: validate generated config with the consumer’s own rules, test the consumer at and beyond its limits, fail safe on bad input by keeping the last good file instead of panicking, and roll out config progressively. OWASP’s new A10:2025 Mishandling of Exceptional Conditions category describes this failure class precisely.
A boundary test for a consumer with a hard limit is a few lines:
#[test]
fn feature_file_over_limit_is_rejected_not_panicking() {
let features: Vec<Feature> = (0..250).map(Feature::dummy).collect();
let result = FeatureSet::try_from_file(&features);
assert!(matches!(result, Err(LoadError::TooManyFeatures { max: 200, got: 250 })));
// and the running config is unchanged
}React2Shell, 3 December
On 3 December the React team disclosed CVE-2025-55182, quickly nicknamed “React2Shell”. It’s a CVSS 10.0 unauthenticated remote code execution vulnerability in React Server Components, caused by unsafe deserialisation of Server Function payloads. Affected react-server-dom-* packages needed upgrading to 19.0.1, 19.1.2 or 19.2.1, and Next.js published patched releases across its 15.x and 16.x lines.
For QA teams the response is a test of process as much as code. Can you list every application using React Server Components? Can you upgrade, run the regression suite and deploy within a day? The SBOM and dependency-inventory practices from our supply-chain essay are what make that possible.
Shai-Hulud 2.0, 24 November
On 24 November a second wave of the npm worm, Shai-Hulud 2.0, compromised hundreds of packages. This time it used a preinstall script that installed the Bun runtime and ran an obfuscated payload. That payload stole local and cloud credentials, published them to public GitHub repositories and used stolen npm tokens to republish more packages. Teams that had set ignore-scripts, scoped CI secrets and a minimum release age after September were in a much better position.
Momentic raises $15M
On 24 November Momentic raised a $15M Series A led by Standard Capital. Momentic lets teams describe critical flows in plain English and uses AI to run them and adapt to UI changes. It’s one more data point in a year of steady investment in AI testing tools.
The Agentic AI Foundation and OWASP’s agentic Top 10
On 9 December the Linux Foundation announced the Agentic AI Foundation, with three founding projects: Anthropic’s Model Context Protocol, Block’s goose agent framework and OpenAI’s AGENTS.md convention for repository instructions to coding agents. MCP went from a launch announcement in November 2024 to neutral, foundation-governed infrastructure in about a year. For testers that’s reassuring: MCP-based testing tooling, including Playwright MCP, sits on a standard that isn’t controlled by a single vendor.
In the same week, the OWASP GenAI Security Project released the OWASP Top 10 for Agentic Applications for 2026:
| ID | Risk | What to test |
|---|---|---|
| ASI01 | Agent Goal Hijack | Injected instructions in content the agent reads |
| ASI02 | Tool Misuse and Exploitation | Agent calling tools outside intended scope |
| ASI03 | Identity and Privilege Abuse | Credentials and permissions granted to agents |
| ASI04 | Agentic Supply Chain Vulnerabilities | Third-party tools, MCP servers, plugins |
| ASI05 | Unexpected Code Execution | Agents generating and running code unsafely |
| ASI06 | Memory and Context Poisoning | Persistent memory corrupted by malicious input |
| ASI07 | Insecure Inter-Agent Communication | Spoofed or tampered messages between agents |
| ASI08 | Cascading Failures | One agent’s error propagating through a system |
| ASI09 | Human-Agent Trust Exploitation | Users over-trusting confident agent output |
| ASI10 | Rogue Agents | Agents acting outside their intended behaviour |
If your organisation is building agents, this list is your security test plan for 2026. If you’re using agents to test, it’s your threat model.
Agents at scale: Spotify’s numbers
In November Spotify’s engineering team wrote about its background coding agent, which had generated more than 1,500 merged pull requests for fleet-wide migrations and code changes. The QA lesson is in how they describe making it work: well-scoped tasks, strong verification and feedback loops. Agents at scale aren’t a replacement for testing. They’re a heavy consumer of it.
2025 in one table
Looking back over this year’s essays, here are the developments that shaped testing practice:
| Month | Development | Why it mattered for QA |
|---|---|---|
| Jan | MCP adoption, OWASP LLM Top 10 2025 | LLM testing gets a shared vocabulary |
| Feb | Operator and browser agents; Reqnroll succeeds SpecFlow | Agents vs scripts; specs matter more |
| Mar | Claude Code preview, Copilot agent mode; Playwright MCP | Agents write tests; MCP connects them to browsers |
| Apr–May | MCP everywhere; k6 1.0 | Agentic tooling explodes; performance testing matures |
| May–Jul | Copilot coding agent, Claude Code GA; vendor AI agents | Agentic QA becomes a purchasing decision |
| Jul–Aug | Replit database deletion; SO survey trust gap; GPT-5 | Evals and guardrails become QA work |
| Aug–Sep | Nx s1ngularity, chalk/debug, Shai-Hulud | Test pipelines are attack surface |
| Oct | Playwright Test Agents; Chrome DevTools MCP | Healing and planning in open-source frameworks |
| Oct–Nov | AWS and Azure outages; cy.prompt; OWASP Top 10:2025 RC | Resilience and exceptional conditions |
| Nov–Dec | Cloudflare; React2Shell; AAIF; OWASP agentic Top 10 | Config safety; agent security; standards consolidate |
What changed for QA engineers
Writing test code stopped being the bottleneck
By the end of 2025, every major framework, IDE and coding agent could generate test code from a description, a recording or an exploration. The Stack Overflow survey showed developers using AI heavily while trusting it less. Generation got cheap. Correct generation didn’t. The scarce resource is now judgement: knowing which tests matter, what they should assert and whether generated output is right.
The oracle became the job
Every agentic tool this year, from Playwright’s planner to cy.prompt to vendor test-creation agents, can drive an application. None of them can know what “correct” means for your business without being told. Expected results, acceptance criteria, boundary values and failure behaviour are where the value is, and they’re classic tester skills.
Quality moved into the pipeline and the platform
The year’s worst incidents weren’t caught by UI tests and couldn’t have been. They needed config validation, boundary tests on consumers, supply-chain controls, progressive rollout and resilience testing. The QA role widened to cover pipelines, configuration and platform behaviour, alongside product features.
Security and QA converged
Supply-chain worms, LLM prompt injection, MCP tool poisoning and agent privilege abuse all landed on QA teams’ desks in 2025, because they’re testable and someone has to test them. OWASP published three lists this year that QA teams can test against: the LLM Top 10, the web Top 10:2025 draft and the agentic Top 10.
What didn’t change
- Testing is still about information. Tests exist to tell people something true about the product. Agents that produce green results without truth make things worse, as we argued in testing vs checking.
- Determinism still gates releases. The most successful agent patterns this year all ended in deterministic checks: CI on agent PRs, explicit assertions after cy.prompt, generated Playwright code reviewed and committed.
- Exploratory testing still finds what scripts miss. Agents can help explore. Deciding what’s worth exploring, and what an odd behaviour means, is still a human skill.
- Communication still matters most. Turning test results into decisions, and risks into priorities, is the part of the job no tool has touched.
Six questions before trusting an AI testing result
As more test results come from agents, healers and AI assertions, testers need a quick way to judge how much weight a result can bear. These six questions work for any tool:
- What exactly was checked? Can you see the assertions, or only a summary saying “passed”? A result you can’t inspect is an opinion.
- Was the oracle human-defined? Did a person decide the expected outcome, or did the tool infer it from the current behaviour of the app? Inferred oracles confirm what the app does, not what it should do.
- Did anything heal or adapt? If locators, steps or waits changed during the run, what changed, and has anyone reviewed it?
- Is it reproducible? Would the same run give the same result? If the tool is non-deterministic, what’s its pass rate over several runs?
- What wasn’t covered? Which flows, data states, browsers or error paths did the run skip? Agents that explore freely rarely report what they didn’t see.
- Who is accountable? If this result is wrong and a defect ships, whose decision was it to trust it?
None of these is new. They’re the questions good testers have always asked about any test result. The difference in 2025 is that more results arrive from systems that sound confident whether they’re right or not. That makes the questions essential rather than optional.
The skills that matter going into 2026
| Skill | Why it matters now | How to build it |
|---|---|---|
| Test design and oracle definition | Agents need expected results | Practise boundary, state and risk-based design; review agent-generated plans |
| Specification writing | Spec-driven agents and planners consume specs | Write acceptance criteria with examples; Gherkin or plain Markdown |
| Reading and reviewing code | Most tests and fixes now arrive as PRs | Review agent PRs; learn your stack’s test frameworks deeply |
| CI/CD and pipeline literacy | Quality gates live in pipelines | Own a pipeline; learn workflow permissions and caching |
| LLM evaluation | AI features need eval suites | Build a golden dataset; learn promptfoo or DeepEval |
| Agent and MCP security | Agents have credentials and tools | Study the OWASP agentic Top 10; run MCP scans |
| Resilience and config testing | Outages come from config and dependencies | Write fault-injection tests; contract-test config |
| Data literacy | Heal logs, flake rates, eval scores | Build a quality dashboard; learn basic statistics |
Team shapes we expect to see more of
Based on the year’s tooling and the practices that worked, three team patterns look likely to grow in 2026. These are trends, not predictions with numbers attached.
- Quality engineering platform teams own test infrastructure, CI gates, agent configuration, MCP servers and eval pipelines that product teams use.
- Embedded quality engineers sit with product teams and focus on specifications, risk analysis, exploratory testing and reviewing agent output.
- AI quality specialists own evals, red teaming and guardrails for AI features, working closely with security.
In each shape, the ratio of writing tests to designing, reviewing and governing tests shifts towards the latter.
A career checklist for 2026
If you’re a QA engineer planning the next year, here’s a practical checklist drawn from the year’s essays:
- Run one agentic workflow end to end, such as Playwright planner, generator and healer, and write up what you had to fix.
- Build an eval suite for one AI feature, with a golden dataset and a CI gate.
- Add one resilience test per critical dependency.
- Implement three supply-chain controls in your test pipeline.
- Review at least ten agent-authored pull requests with a written checklist.
- Map your organisation’s AI features or agents against the OWASP agentic Top 10.
- Present a quality dashboard to your engineering leadership with flake rate, escape rate and heal rate.
Pros and cons of the AI-era QA role
Pros
- Higher leverage: one tester can direct far more automated checking.
- More interesting work: design, risk, security and systems thinking.
- A clearer business case, because AI-generated change needs verification.
- Open standards (MCP, AGENTS.md) and open-source agents lower the barrier to experimenting.
Cons
- Pressure to equate “AI generated a lot of tests” with “we’re well tested”.
- Skills shift quickly, and training budgets lag.
- Accountability is blurred when agents author, heal and approve.
- New security responsibilities without always having new authority.
What this means for your team
Redefine success metrics. Stop counting tests written. Track escaped defects, time to detect, flake rate, mutation score on critical modules, eval pass rates and heal review rates.
Make the oracle explicit. Invest in specifications, acceptance criteria and golden datasets. They’re what make every agent useful.
Own the gates. QA should co-own CI quality gates, agent permissions and healing policy. If agents can merge, the gate is the quality strategy.
Test config, limits and failure. The year’s outages say it plainly: generated config, hard limits and dependency failures need tests as much as features do.
Bring security into the test plan. Use the three OWASP lists of 2025 as test catalogues for web apps, LLM features and agents.
Invest in people. Fund training in evals, agent security, CI and code review. Tools change every month. Testers who can judge them stay valuable.
2025 was the year testing tools learned to drive applications by themselves. 2026 will be about whether teams can trust what those tools report. That trust is built by people who design the checks, question the results and own the decision to ship. In other words, by testers. For the tools that defined this year, see Best AI testing tools 2025 and Best web testing tools 2025.