Google Artemis Review: AI Agent for Android Testing

Mobile testing has been waiting for its Playwright MCP moment. Web teams have had agent-friendly browser tooling for more than a year, while Android testers have mostly been wiring LLMs to ADB scripts themselves. In August 2026, Google published Artemis on GitHub, an open-source agent that “turns natural-language instructions into reliable Android automation” on real phones and emulators.

This review covers what Artemis does, how its Flash and Pro execution profiles work, how it plugs into AI coding assistants through MCP, what its benchmark claims mean, and the attribution dispute with Minitap that surfaced in September. It reflects the repository as of mid-September 2026.

What is Google Artemis?

Artemis is an Apache-2.0 licensed Python project in the google GitHub organisation. Its README tagline is “Let AI assistants and test suites use real phones like a human.” The repository’s first commit, on 13 August 2026, is titled “initial release of ARTEMIS Android Agent & Automation Platform.”

The README lists these key features:

• Cross-app automation from natural-language instructions.
• Multimodal targeting using element indices when available, with coordinate and visual fallbacks for custom interfaces.
• IDE diagnostics through a Model Context Protocol server, so tools like Antigravity, Claude Code and Windsurf can drive test devices and collect Logcat output and screenshots.
• Flash execution, a reactive loop that is “typically 3–5s per step.”
• Pro exploration, which checks targets before actions and supports long-running exploratory and stability tests.
• AndroidWorld results of “99%+ task completion” on Google Research’s benchmark.

Four ways to use Artemis

The README describes four usage modes:

1. Web console (uv run artemis ui) with live screen mirroring, natural-language task dispatch, reasoning telemetry, action trajectories and replays.
2. MCP server for AI IDEs and agents, aimed at bug reproduction and test execution.
3. Developer CLI (uv run artemis run) for automated cases, exploratory stability checks and benchmarks.
4. Python SDK for integrating into frameworks such as pytest or CI pipelines, with “strongly typed Pydantic structured outputs.”

Quick start

The project expects an Android device with USB debugging enabled, or an emulator. A start script detects and installs ADB, scrcpy, FFmpeg and Python dependencies through uv:

git clone https://github.com/google/artemis.git && cd artemis
./start.sh

A one-off task from the CLI looks like this:

uv run artemis run "Open Settings, find Battery and tell me current level" --profile flash

The Python SDK in a test

The SDK example in the README is a good template for a CI-friendly check:

import asyncio
from artemis_client import ArtemisClient

async def main():
    client = ArtemisClient(
        "http://artemis-host:8000",
        device_serial="emulator-5554",
        default_profile="flash",
    )
    result = await client.run(
        "Open System Settings, go to 'Battery', verify battery percentage "
        "is displayed, and check for any crash dialogs.",
    )
    assert result.succeeded, f"Test failed: {result.error or result.status}"

asyncio.run(main())

The client package is described as zero-runtime-dependency. ADB, agents, models and image processing stay on the device host, which means a CI runner can talk to a shared device machine over HTTP.

Flash vs Pro: two execution profiles

This is the most interesting design choice in Artemis, and it maps neatly to how testers already think about smoke tests versus exploratory sessions.

Profile	How it works	Speed	Best for
Flash (--profile flash)	One model observes the screen, thinks and acts in a reactive loop	~3–5 s per step	Routine, deterministic UI tasks
Pro (--profile pro)	Multi-agent graph: a Planner keeps a Markdown plan with verify and assert items, an Operator executes with pre-execution checks	~15–40 s per step	Planning, verification, long exploratory runs

In Pro mode, every action passes a pre-execution check against the live UI tree and pixels, and blocked actions return to the Operator for recovery. Both profiles share a compressed history in which older screenshots are replaced by visual summaries and completed steps are grouped into searchable chunks.

That history compression is worth noting. Slack’s agentic testing study, which we covered in Agentic vs deterministic E2E tests, found that token cost is driven by context growth across turns. Artemis’s design explicitly targets that problem, although I haven’t seen published cost figures for it.

How Artemis finds elements

Android UIs are harder for agents than web pages. There is no DOM, and apps built with Canvas, Jetpack Compose or Flutter may expose little to the accessibility tree. Artemis combines accessibility hierarchies and OCR with visual models for these custom interfaces.

The MCP rules file shipped with the project describes a “Dynamic-First, Coordinate-Fallback” locator pattern. In practice that means structured elements first, coordinates only when nothing else works.

The on-device helper

On first use, Artemis installs an “Artemis Accessibility Helper,” a small accessibility service that reads screen layout. The README says it “listens only on the phone itself and sends nothing elsewhere.” You’ll see a notification and a new entry under Accessibility settings.

Testers should know about this for two reasons:

• It changes the device state. Some apps behave differently when accessibility services are active. If you test apps with accessibility-sensitive behaviour, compare results with the helper on and off.
• It can be disabled. Setting ARTEMIS_HIERARCHY_BACKEND=uiautomator uses UIAutomator2 instead, and ARTEMIS_HELPER_AUTO_INSTALL=false prevents automatic installation. If the helper fails mid-task, Artemis falls back to UIAutomator2 and reports it.

MCP integration with AI coding assistants

Artemis ships an MCP server with tools such as mobile_run_task, mobile_manage_task, mobile_get_device_state, mobile_inspect_trace and mobile_diagnose. It can install configuration for several clients:

uv run artemis mcp --install all

The README lists Antigravity, Cursor, Claude Code, Codex, Windsurf, VS Code, Cline and Roo, and OpenClaw among supported IDEs. It also offers a “testing mindset” rules file covering active exploration before coding, routing between Flash and Pro, and latency compensation.

The README’s suggested prompt shows the intended workflow: build the latest changes into an APK, install it, open the login screen with a test account, check for unexpected popups and return screenshots. That is a developer verifying their own change on a real device without leaving the IDE, which is exactly the inner-loop use case Playwright MCP serves on the web.

The AndroidWorld benchmark claims

AndroidWorld is a Google Research benchmark the README describes as “spanning 20+ apps and 100+ multi-step tasks.” Artemis claims 99%+ completion.

Treat that number carefully:

• It is self-reported. Minitap notes that the AndroidWorld leaderboard “explicitly says it does not independently verify” submitted results.
• It is a general-purpose task benchmark. Completing “send a message” or “create a calendar event” is not the same as finding bugs in your app.
• Competitors claim similar scores. Minitap says it submitted 94.8% and then 100% for its own agent, results that were not reflected on the leaderboard as of 11 September.

A high AndroidWorld score is good evidence that an agent can operate a phone. It says little about whether that agent will notice a wrong total on your checkout screen. Only your own evaluation can tell you that.

The Minitap attribution dispute

On 11 September 2026, Minitap CEO Nicolas Dehandschoewercker published I expected better from Google. Minitap built mobile-use, an Apache-2.0 open-source agent for controlling phones.

The post claims that:

• Parts of Artemis’s Android device connection code “matched our implementation exactly.”
• The instructions for an agent called Hopper were “identical, word for word.”
• An example task, sending Happy New Year messages to Alice, Bob and Charlie, was the same.
• Older versions shared a bug in a results helper.
• An earlier Artemis pyproject.toml listed three Minitap authors, and a later version replaced them with another author via a force push in August.
• The README did not credit mobile-use.

Minitap published links to the specific files and a public record of its comparison. I haven’t independently reviewed the code, so these are Minitap’s claims.

What the Artemis commit history does show is a response. On 12 September 2026 (UTC), a commit titled “complete README and relevant file headers per Apache 2.0 requirements” landed, and the README’s License section now says: “This project includes source code developed by Minitap, Inc.” Software Testing Weekly summarised it as Google adding “the missing attribution” a day after the post.

Why testers should care

This might look like open-source politics, but it matters for tool selection:

• Licence compliance: if you redistribute or modify Artemis internally, your legal team will want the provenance clear. It is now documented in the README.
• Project maturity: a repository that was weeks old in mid-September is still settling. Expect API and configuration changes.
• Alternatives exist: mobile-use is open source too, and comparing both on your app is reasonable.

Artemis vs traditional Android test frameworks

	Artemis	Espresso / UIAutomator	Appium	Maestro
Test definition	Natural language	Kotlin or Java code	Code in many languages	YAML flows
Determinism	Low to medium	High	High	High
Speed per step	3–40 s	Milliseconds	Sub-second to seconds	Sub-second to seconds
Handles UI changes	Adapts	Needs code changes	Needs locator changes	Needs flow changes
Cross-app flows	Yes	Limited	Yes	Yes
Best use	Exploration, bug reproduction, smoke checks from IDE	Fast in-app regression	Cross-platform regression	Readable E2E flows

Artemis doesn’t replace Espresso, Appium or Maestro. It sits alongside them, in the same place Slack put agentic testing: a layer on top of deterministic tests, useful for exploration and debugging.

Where Artemis fits in a mobile test strategy

Here’s how I’d pilot it.

1. Inner loop first. Connect the MCP server to your AI assistant and use it to verify changes on a device before opening a pull request.
2. Bug reproduction. Feed it bug reports in plain language and see whether it reproduces them, capturing Logcat and screenshots.
3. Exploratory stability runs in Pro mode on new features.
4. Smoke checks in CI with Flash mode on a small number of critical flows, measuring flakiness and run time over a few weeks.
5. Convert findings into deterministic tests in Espresso or Maestro for fast regression.

Measure success rate, time per run and model cost on your own app. Don’t rely on benchmark scores.

Security and privacy checks before a pilot

An agent that drives a real phone sees whatever is on its screen. Before connecting Artemis to anything beyond a test emulator, work through a short checklist with your security team.

• Model provider: Artemis is multimodal, and the README’s badges list Gemini, Claude, GPT-4o and Qwen-VL. Whichever model you configure will process screenshots of your app. Confirm your data-handling terms cover that.
• Test accounts only: use dedicated accounts with synthetic data. Never point an agent at a personal phone or a production account.
• USB debugging: devices with debugging enabled should live in a controlled device lab, not on desks or shared networks.
• Cross-app access: Artemis can operate across apps. Remove apps from test devices that the agent has no reason to open, such as email or messaging with real accounts.
• Logs and replays: Logcat output, screenshots and recordings may contain tokens or personal data. Decide where they’re stored and for how long.
• MCP scope: an IDE assistant with the Artemis MCP server installed can control the device. Install it only in workspaces that need it.

None of this is unusual for mobile test labs. Agents simply make it easier to forget, because a one-line prompt can trigger a long sequence of actions.

How to measure a pilot

Pick five to ten flows that matter, including at least one built with a custom UI toolkit such as Compose or Flutter, and run each one 10 times in Flash and Pro modes. Record:

Metric	Why it matters
Success rate per flow	Reliability on your app, not a benchmark
Median and worst run time	Whether it fits in CI or only on demand
Model cost per successful run	Budget at scale
False passes	Runs that reported success but missed a visible problem
Fallbacks used	How often coordinate or visual targeting was needed
Bugs found that existing tests missed	The real value

False passes deserve the most attention. An agent that confidently reports success on a broken screen is worse than a test that fails loudly. Seed a few known bugs into a test build and check that Artemis catches them, the same idea as the mutation testing approach for unit tests.

Limitations to know

• Android only. iOS is on the roadmap, not available.
• Model costs and latency apply to every step. Pro mode at 15 to 40 seconds per step is not for large regression suites.
• Very young project. Weeks old in mid-September, with frequent changes.
• Device state changes from the accessibility helper.
• Non-deterministic paths, as with any agent.
• Benchmark scores are self-reported.

The roadmap lists Android Studio integration, iOS expansion, on-device lightweight vision models and real-time voice interaction.

Verdict

Artemis is the most complete open-source attempt so far to give AI coding assistants real Android devices. The Flash and Pro split is a sensible answer to the speed-versus-rigour trade-off, the MCP server and SDK cover both IDE and CI use, and the fallbacks for Compose and Flutter interfaces address a real pain point.

It is also very new, its benchmark claims are self-reported, and its launch came with an attribution dispute that has since been addressed in the README. Pilot it for exploration and bug reproduction, keep your deterministic suite, and let your own data decide how far to take it.

Frequently asked questions

What is Google Artemis?

Artemis is an open-source, Apache-2.0 licensed AI agent from Google’s GitHub organisation that turns natural-language instructions into Android automation on real devices and emulators, with an MCP server, CLI, web console and Python SDK.

Is Artemis free?

The code is open source under Apache 2.0. You still pay for the AI models it uses and provide your own devices or emulators.

Does Artemis support iOS?

Not as of September 2026. iOS expansion is on the project’s roadmap.

What is the difference between Flash and Pro in Artemis?

Flash is a fast reactive loop at about 3 to 5 seconds per step for routine tasks. Pro is a multi-agent planning and verification workflow at about 15 to 40 seconds per step for more complex or exploratory work.

Did Google copy Minitap’s mobile-use code?

Minitap’s CEO publicly claimed on 11 September 2026 that Artemis reused mobile-use code without credit. On 12 September, an Artemis commit added an attribution to Minitap in the README. I haven’t independently verified the code comparison.

Can Artemis replace Espresso or Appium?

No. Agents are slower and less deterministic. Artemis is best used alongside deterministic frameworks for exploration, bug reproduction and IDE-driven checks.

Further reading on CheckOps

• Agentic vs deterministic E2E tests: lessons from Slack
• Playwright CLI vs Playwright MCP
• Vibium review: AI-native browser automation
• Browser agents vs test automation
• Best AI testing tools 2026