AI Testing Agents Arrive: Computer Use and DORA 2024

On a single day, 22 October 2024, three things landed that together define where AI testing stands at the end of the year. Anthropic released a model that could operate a computer by looking at the screen. Google published the 2024 DORA report, which found that more AI adoption was associated with less stable software delivery. And Capgemini’s World Quality Report said two-thirds of organisations were already using or planning generative AI for quality engineering.

Taken together, those three findings capture the moment. The capability is real and arriving fast, adoption is broad, and the evidence that it improves outcomes is, so far, mixed. This essay covers mid-October to mid-November 2024 and sets out a practical map of AI testing tools: what each category does, where it helps, and where it hurts.

Computer use: the first general-purpose UI agent

Anthropic’s upgraded Claude 3.5 Sonnet introduced “computer use” as a public beta. Through the API, the model receives screenshots, reasons about what’s on screen, and returns actions: move the mouse, click, type, scroll. A developer-supplied harness executes those actions and sends back the next screenshot.

Anthropic was candid about the limits. On the OSWorld benchmark, which measures an agent’s ability to complete real computer tasks, the model scored 14.9% in the screenshot-only category. That was well ahead of the next-best system’s 7.8%, and rose to 22.0% when allowed more steps. Humans score far higher. Anthropic described the capability as “at times cumbersome and error-prone”.

For testers, the headline isn’t the benchmark. It’s the interface. Until now, AI testing tools worked through the DOM: they read HTML, generated selectors and drove a browser through Selenium, Playwright or a proprietary engine. Computer use works through pixels, like a human would. That has three implications:

1. It’s application-agnostic. The same agent can drive a web app, a desktop app, a legacy thick client or a remote desktop session. For organisations with large estates of hard-to-automate software, that’s new.
2. It’s slow and expensive per step. Every action costs a screenshot round-trip through a large model. Nobody is going to run a 2,000-test regression suite this way in 2024.
3. It’s non-deterministic. The same instruction can produce different action sequences on different runs. That’s useful for exploration and a problem for regression.

The honest assessment: computer use is a research-grade capability that points at the future of exploratory and end-to-end testing. It doesn’t replace your Playwright suite. It might, eventually, replace the parts of manual regression that nobody has automated because the app is too awkward to automate.

DORA 2024: the stability warning

The 2024 DORA report is the most rigorous large-scale look yet at what AI adoption does to software delivery. Its headline is easy to misread, so here it is precisely. DORA estimates that a 25% increase in AI adoption is associated with a 1.5% decrease in delivery throughput and a 7.2% decrease in delivery stability.

That isn’t “AI makes software worse”. The same report found individual developers reporting higher productivity, flow and job satisfaction with AI. The report’s own hypothesis is that AI makes it easy to produce larger changes, and larger changes are riskier. DORA has shown for years that small batch sizes and robust testing are what keep delivery stable. AI tools that inflate batch size undermine exactly those practices.

For QA, this is the most important finding of the year. It says:

• AI-assisted coding raises the volume and size of changes reaching your pipeline.
• If your test and review capacity doesn’t scale with it, stability drops.
• The answer isn’t to ban AI. It’s to keep batches small, invest in fast automated checks, and treat AI-generated code as code that needs more verification, not less.

We expand on this in our March 2025 essay on AI coding agents writing tests.

World Quality Report 2024-25: adoption is broad

Capgemini, Sogeti and OpenText’s World Quality Report 2024-25, published the same day, reported that 68% of organisations were either actively using generative AI in quality engineering (34%) or had roadmaps after successful pilots (34%). Only 4% said they weren’t exploring it at all, down from 31% the year before. 72% reported that GenAI was helping them automate faster.

Survey data like this measures intent and perception more than outcomes, and should be read alongside DORA’s harder delivery metrics. But together they make one thing clear: by the end of 2024, “should we use AI in testing?” is the wrong question. The useful questions are where, how, and with what safeguards.

GitHub Universe: model choice comes to Copilot

On 29 October, at GitHub Universe, GitHub announced multi-model Copilot, with Anthropic’s Claude 3.5 Sonnet, Google’s Gemini 1.5 Pro and OpenAI’s o1-preview and o1-mini joining GPT-4o in Copilot Chat. For test automation this matters more than it looks. Different models are better at different tasks, and teams can now pick a model for test generation separately from the one they use for refactoring or explanation. It also signals that the assistant layer is becoming a commodity. The differentiation is moving to context, integration and workflow.

What a computer-use test loop looks like

It helps to see the shape of an agentic test loop, because it explains both the promise and the cost. Stripped of details, every computer-use harness looks something like this:

goal = "Log in as the demo user and add the blue mug to the cart. Report the cart total."
history = []

for step in range(MAX_STEPS):
    screenshot = vm.screenshot()
    action = model.next_action(goal=goal, screenshot=screenshot, history=history)
    if action.type == "done":
        report = action.summary
        break
    vm.execute(action)            # click(x, y), type(text), scroll(...)
    history.append(action)
else:
    report = "gave up after MAX_STEPS"

assert "cart total" in report.lower()

Three things stand out. First, there’s no selector anywhere. The agent finds the login button the way a person would, which is why it survives UI redesigns that break scripted tests. Second, every iteration is a model call with an image, so a 25-step journey is 25 multimodal requests. That’s seconds to minutes of latency and real money per run. Third, the assertion is weak. The agent tells you what it thinks happened. Turning that into a trustworthy verdict needs a separate oracle: an API check of the cart contents, a database query, or at least a structured output schema.

That third point is the real research problem for agentic testing over the next couple of years. Agents can do things. Verifying that the right thing happened, deterministically and cheaply, is still a job for conventional automation.

The cost math nobody puts in the demo

Before running agents at any scale, do the arithmetic. A rough per-run cost for an agentic end-to-end check is:

cost_per_run ≈ steps × (input_tokens_per_step × input_price + output_tokens_per_step × output_price)

Screenshots are token-heavy, and context grows as history accumulates. So per-step cost rises through a run, and long journeys are disproportionately expensive. Compare that with a scripted Playwright test, whose marginal cost is a few seconds of CI compute. Agents start to make economic sense only where the alternative is human time: manual regression passes, exploratory sessions, or applications that would take weeks to automate conventionally. For everything else, a hybrid works better. Use agents to draft or repair scripted tests, then run the scripts.

Governance: questions to settle before the pilot

AI testing tools touch production-like data, credentials and sometimes real customer environments. Before a pilot starts, settle a few governance questions with security and legal:

• Data exposure. Which screens, logs and payloads will be sent to a third-party model? Are there PII or regulated fields on those screens?
• Credentials. Agents need logins. Use dedicated test accounts with least privilege, never shared or personal credentials.
• Environment isolation. Agents that can click anything will eventually click “Delete all”. Run them against disposable environments.
• Auditability. Keep the action log and screenshots for every agent run, so a surprising result can be reconstructed.
• Vendor model changes. Ask how you’ll be notified when the underlying model changes, since behaviour can shift overnight.

None of these is a reason not to experiment. They’re the difference between a pilot that informs a decision and one that becomes a security incident.

A map of AI testing tools in late 2024

“AI testing tools” covers several very different product categories. Lumping them together is why so many evaluations go wrong. Here’s the map we use:

Category	What it does	Representative tools	Determinism
Assistive code generation	Writes test code from prompts or source; human reviews and commits	GitHub Copilot, Cursor, Qodo, Codeium	Output is plain code, so runs deterministically
Self-healing locators	Repairs broken selectors at runtime using multiple attributes or ML	Testim, mabl, Healenium, Functionize	Mostly deterministic, with fallbacks
Natural-language / codeless	Tests written in plain English, interpreted by the platform	testRigor, mabl, KaneAI, Katalon	Varies; often cached interpretations
Visual AI	Compares screenshots semantically, not pixel-by-pixel	Applitools, Percy, Chromatic	Deterministic comparison with AI filtering
Managed AI + humans	Vendor builds and maintains your suite with AI tooling and staff	QA Wolf, Rainforest QA	Output usually Playwright or proprietary code
Autonomous agents	Agent explores or executes goals via DOM or pixels	Claude computer use, early open-source agents	Non-deterministic

Assistive code generation

This is where most teams start, and it’s the safest category because the output is ordinary test code that goes through review. The risk is subtle. Generated tests often assert what the code does rather than what it should do, so they lock in bugs. A test generated from a buggy function will pass, and keep passing.

A practical guardrail is to generate tests from the specification rather than the implementation, and to review assertions first:

Prompt pattern that works better than "write tests for this file":

"Here is the acceptance criteria for the discount rules (pasted below).
Write Vitest unit tests for applyDiscount() that cover each rule,
the boundaries between rules, and invalid inputs.
Do not read the implementation. List the cases before writing code."

Self-healing locators

Self-healing tools record several attributes for each element: ID, classes, text, position, neighbours. When the primary locator fails, they pick the best match. They genuinely reduce maintenance on UIs that change often. The risk is silent false passes. If a “Delete” button is healed to a nearby “Archive” button, the test goes green while testing the wrong thing. Any self-healing tool you adopt should log every heal, and someone should review those logs. We go deep on this in our October 2025 essay on self-healing tests.

Natural-language and codeless platforms

These platforms let product managers, manual testers and domain experts write tests such as “click Checkout, then verify the total includes tax”. The good ones are excellent for teams without strong automation engineers. The trade-offs are vendor lock-in, harder version control and diffs, and costs that scale with usage. Ask every vendor how tests are stored, whether you can export them, and what happens to your suite if you leave.

Visual AI

Visual testing is arguably where AI in testing delivered value earliest. Applitools’ Visual AI, Percy and Chromatic reduce the noise of pixel diffs caused by anti-aliasing, dynamic content and rendering differences. In July, Applitools added natural-language authoring with Autonomous 2.0, blurring the line between visual and functional platforms.

Managed services

QA Wolf, which raised $36M in July, combines AI tooling with human engineers and sells coverage as a service. For teams that need coverage quickly and can’t hire, this model makes sense. The question to ask is what you own at the end of the contract.

Autonomous agents

This is the frontier computer use opened up. Agents can explore an app, attempt goals and report what broke. In late 2024 they’re best suited to exploratory testing assistance and smoke checks on hard-to-automate apps, not regression. Our February 2025 essay on browser agents looks at how this category develops.

Where AI testing helps today, and where it doesn’t

Strong fit in late 2024:

• Generating unit and API test scaffolds that engineers then refine.
• Converting manual test cases into automation drafts.
• Generating varied, realistic test data.
• Reducing locator maintenance on fast-changing UIs, with heal logging.
• Visual regression with semantic diffing.
• Summarising test failures and clustering flaky tests.

Weak fit in late 2024:

• Deciding what to test. Risk analysis still needs domain knowledge.
• Writing oracles, the “what’s the correct answer?” part, for complex business logic.
• Replacing deterministic regression suites with agents.
• Security and compliance-critical verification where every step must be auditable.

A simple evaluation rubric

When a vendor demo looks magical, these questions bring it back to earth:

Question	Why it matters
What’s the false pass rate?	A tool that heals into wrong elements is worse than one that fails
How are tests stored and versioned?	Lock-in and code review
What does a failure report contain?	Time to diagnose dominates cost
How does it behave on your app?	Demos use cooperative apps
What happens on a model update?	Non-determinism across vendor model changes
What does it cost at 10× usage?	Per-run pricing scales with success

Run every candidate against the same three flows from your real product, one of them deliberately awkward, for at least two weeks in CI.

A note on LLM testing

There’s a mirror-image trend worth flagging: testing of AI, not testing with AI. Teams shipping LLM features need evaluations, red-teaming and regression checks for prompts and models. Open-source tools like promptfoo and DeepEval have grown quickly this year. We cover LLM testing properly in our January 2025 essay on MCP and the OWASP LLM Top 10 and in the August 2025 essay on LLM evals.

What this means for your team

Adopt AI where the output is reviewable. Start with assistive generation of unit and API tests, reviewed like any other code. That’s the highest-value, lowest-risk entry point.

Protect batch size. DORA’s finding is a warning about process, not tools. If AI-assisted development is inflating pull requests, set soft limits, encourage stacked PRs, and make sure your fast test layers can keep up.

Log every heal. If you use self-healing locators, treat each heal as a signal to review, not a silent fix.

Pilot agents on the awkward corners. If you have a legacy desktop app or an internal tool nobody automates, that’s a reasonable place to experiment with computer-use style agents, as exploration aids rather than release gates.

Measure outcomes, not adoption. Track change failure rate, escaped defects and time to diagnose before and after introducing AI tools. Survey enthusiasm isn’t the same as delivery improvement. DORA’s numbers show the two can diverge.

Invest in the skills that don’t automate. Risk analysis, test design, oracle definition and system thinking matter more when generating test code is cheap. We’ll return to what that means for the QA role in our 2025 year-in-review.

October 2024 will probably be remembered as the month AI agents became tangible for testers. The right response is neither hype nor dismissal. It’s careful, measured experimentation, with the same scepticism we apply to any test that claims to pass.

Further reading on CheckOps

• Playwright vs Cypress vs Selenium: the BiDi shift
• Shift-left meets shift-right: Sonos and observability
• MCP and the OWASP LLM Top 10: LLM testing starts here