Essay
Agentic vs Deterministic E2E Tests: Lessons from Slack
Slack ran 200+ agentic E2E workflows with Playwright MCP, Playwright CLI and generated tests. What its failure rates, costs and token data mean for testers.
15 July 2026 · 11 min · Irfan Ahmad
Most of the debate about AI agents in end-to-end testing has been opinion. In June 2026 Slack’s engineering team published something rarer: data. In Agentic Testing: Where Agents Fit in the E2E Testing Stack, staff engineer Sergii Gorbachov describes more than 200 agentic E2E runs comparing three approaches on the same flows, with failure rates, run times, token counts and costs.
This essay summarises what Slack found, where I think the results generalise, and how to use them to decide between agentic and deterministic end-to-end tests. It is a follow-up to our comparison of Playwright CLI vs Playwright MCP, and Slack’s data adds an important twist to that story.
The core idea: journeys vs goals
The post opens with a clean distinction.
- A traditional E2E test validates a journey:
click → click → type → assert. - An agent-driven test validates a goal:
goal → agent adapts → verify result.
Slack’s one-line summary is worth memorising: “Tests enforce journeys. Agents verify goals.”
In Slack’s runs the overall workflow stayed consistent, for example login, search, result, clear, but the exact actions varied. Agents clicked a search suggestion one time and pressed Enter the next, reused existing state or reopened search, and added or skipped intermediate steps. That flexibility is the point of agentic testing, and also the source of its costs.
Slack’s experiment
Three execution models
- Agent + Playwright MCP: the agent uses predefined browser actions with persistent context such as DOM snapshots and logs.
- Agent + Playwright CLI: the agent runs CLI commands through the shell, one step at a time.
- Generated Playwright tests: an agent writes deterministic Playwright test code from a natural-language description, runs it and refines it until it passes.
Setup
| Item | Value |
|---|---|
| Agent model for MCP and CLI | Claude Sonnet 4.5 |
| Model for generated tests | Claude Opus 4.6 |
| Execution | Non-interactive Claude Code (claude -p) |
| Environment | Test workspaces with non-production data, plus a Slack Dev API MCP |
| Runs | 20 per configuration, more than 200 in total |
Two flows
- Thread Reply (simple): about 15 to 20 steps. Create a channel, send a message, reply in a thread, verify thread state.
- Search Discovery (medium): about 25 to 30 steps. Enter searches, navigate results, move between views, verify outcomes.
Agent approaches were tested with two input formats: detailed natural-language instructions and the same workflow expressed as structured YAML.
The results
Slack’s summary table:
| Approach | Failure rate (thread reply) | Failure rate (search discovery) | Avg runtime |
|---|---|---|---|
| Agent (Playwright MCP) | 0% | ~12% | ~5–8 min |
| Agent (Playwright CLI) | ~12% | ~20% | ~9–11 min |
| Generated Playwright tests | ~8% | ~48% | ~3 min |
Reliability
Playwright MCP was the most reliable agent configuration, near zero on the simple flow and within 0 to 12% overall. The CLI was worse, at roughly 12 to 20%, and Slack says most CLI failures came from “authentication and navigation issues (sign-in errors, timeouts, and session instability), suggesting that many failures were caused by the execution layer rather than the agent’s reasoning.”
Generated tests did reasonably on the simple flow but failed about 48% of the time on the complex one. Slack is careful here: these tests “typically progressed through 70-80% of the flow before breaking on a final interaction or assertion,” and were generated from loosely specified descriptions while reusing existing page objects, which sometimes interfered with precise element targeting.
Slack’s explanation for MCP’s edge is state. “MCP keeps a live, stable view of the app, while CLI rebuilds state from snapshots at each step.” The authors also suspect the MCP agent reuses successful interactions from earlier in the session, though they didn’t measure it.
Speed
Generated tests were the fastest at about 3 minutes, and that includes generation. Each test was generated once and run five times. Raw execution was about 32 seconds for thread reply and 45 seconds for search discovery. As Slack notes, in CI “the one-time generation cost becomes negligible.”
Agents pay their cost on every run, because every step involves observing, reasoning and acting.
Adaptability
Only about 20% of agent runs followed exactly the same sequence of actions, even after normalising away parameters, waits and equivalent tool variants. Most runs found different valid paths to the same goal.
Cost
Agent-driven runs typically cost $15 to $30 per execution. Token use on the search discovery flow:
| Approach | Tokens | Turns |
|---|---|---|
| MCP (Opus 4.6) | ~3.8M | ~40 |
| MCP (Sonnet 4.5) | ~3.5M | ~40 |
| MCP (Haiku 4.5) | ~5.7M | ~60 |
| CLI (Opus 4.6) | ~6M | ~85 |
| Code Gen (Opus 4.6) | ~7M | ~70 |
Slack’s conclusion: “how the agent was executed mattered more than which model powered it.” Because the underlying API is stateless, every turn re-sends the system prompt and the whole conversation. Cost is driven by turn count and context growth, and “the majority of the cost was retransmission of previously seen content.”
The surprise: CLI used more tokens than MCP
In February the Playwright team described the CLI as more token-efficient than MCP, because it avoids loading tool schemas and accessibility trees into context on every step. We repeated that reasoning in our CLI vs MCP comparison.
Slack’s data shows that the per-step saving can be wiped out by more steps. The CLI took about 85 turns compared with MCP’s 40 to 60, “because each browser interaction was split across multiple commands, such as actions, waits, snapshots, reads, and element lookups.” MCP “combined interaction and state return into a single round trip.” Each extra turn re-sends everything.
Both things can be true. A single CLI call is lighter than a single MCP call. A whole flow can still be heavier if the agent needs twice as many calls. The lesson is to measure total tokens per completed task on your own flows, not cost per call.
Slack also notes that its token use was not optimised, and lists prompt caching, context compaction and reducing snapshot frequency as opportunities.
Infrastructure matters as much as the model
The post’s most practical finding may be about the execution environment. CLI failures clustered around authentication, navigation timing and session handling. MCP runs were easy to parallelise; CLI runs “were difficult to parallelize in our setup and were mostly executed sequentially.”
If you are evaluating agentic testing, budget time for the boring parts: stable test accounts, reliable login, deterministic test data and sane timeouts. Many apparent “AI failures” are environment failures.
Where agentic testing fits: a new layer on the pyramid
Slack’s answer is not replacement. It adds a layer.
Deterministic E2E tests
“Best suited for fast, repeatable regression checks in CI.” They can be human-written or AI-generated, they are cheap to run, and they enforce a specific journey.
Agentic testing
Agents work from a goal, observing the UI, reasoning about state and deciding how to reach the outcome. Slack lists three uses:
- Exploring complex UI behaviour
- Debugging flaky workflows
- Reproducing production bugs
The resulting pyramid has four layers: unit, integration, E2E, and agentic testing on top. Slack stresses that agentic tests still operate at the same level as E2E tests, through the UI; the difference is how they run. In their words, “Deterministic tests provide a stable foundation for CI, while agentic testing adds a distinct layer at the top of the testing pyramid.”
On cost, Slack is direct: “agent-driven testing may currently be better suited for targeted debugging or exploratory testing than for high-frequency CI execution, although cost may improve with future models and tooling.”
Limits of the study
The authors are candid about scope, and so should we be.
- Two flows, one product. Results on Slack’s UI may not match yours.
- Single-session workflows only. Cross-workspace flows and multi-window scenarios were not explored.
- Unoptimised token use. Costs could fall with caching and compaction.
- Specific models and dates. Model capabilities change quickly.
- Generated tests from loose descriptions. Better specifications would likely improve the 48% failure rate on complex flows.
None of these undermine the direction of the findings. They do mean you should repeat a small version of the experiment before making big decisions.
How to run your own version
A one-week experiment can tell you a lot.
- Pick two flows, one simple and one complex, that matter to your product.
- Prepare a stable environment: dedicated test accounts, seeded data, reliable authentication.
- Run each approach 10 to 20 times: agent with MCP, agent with CLI, and generated deterministic tests.
- Record outcome, duration, turns and tokens for each run.
- Classify failures as reasoning, environment or application bugs.
- Calculate cost per successful run, not cost per run.
- Decide placement: which flows belong in CI as deterministic tests, and which are better explored by agents on demand.
Writing goals an agent can verify
An agent can only verify the goal you give it. Slack tested both detailed natural-language instructions and structured YAML, and noted that “the difference is not the level of detail, but how that detail is represented.” Whichever format you choose, a few habits help.
- State the end condition, not just the action. “Reply in a thread” is an action. “Reply in a thread, and verify it appears in All Threads” is a goal with a check, which is the example Slack itself uses.
- Name the data. Give the agent the channel name, the message text and the account to use, so a pass means something specific.
- Say what must not happen. If an error toast, a duplicate message or a console error should count as a failure, write that down.
- Keep one goal per run. Long, multi-goal prompts grow context and turn count, which Slack showed are the main cost drivers.
- Record the path. Keep the action log for every run. When only about 20% of runs follow the same sequence, the log is how you understand what was actually tested.
A structured format such as YAML makes these fields explicit and easier to review in a pull request. Natural language is quicker to write for one-off exploration. Many teams will end up using both.
Keeping agentic test costs under control
Slack’s numbers came from runs where token use “was not optimized,” and the post lists prompt caching, context compaction and reducing snapshot frequency as ways to cut cost. In practice that suggests a few rules for anyone piloting agentic tests:
- Set a budget per run and stop runs that exceed it, rather than letting a confused agent loop.
- Track turns as well as tokens. Turn count was the clearest predictor of cost in Slack’s data.
- Trigger agents on demand, for a new feature, a bug report or a flaky test, instead of on every commit.
- Convert stable findings into code. Once an agent has reproduced a bug or found a reliable path, a generated deterministic test runs in seconds. Slack’s generated tests executed in about 32 to 45 seconds once written.
The economics may change as models and tooling improve, as Slack notes. For now, treating agent runs as an investigation tool rather than a regression suite keeps the bill predictable.
Decision guide: agentic or deterministic?
| Situation | Use |
|---|---|
| Critical regression check on every pull request | Deterministic E2E |
| Smoke test after each deploy | Deterministic E2E |
| Exploring a new or complex feature | Agentic |
| Reproducing a bug report | Agentic, then convert to a deterministic test |
| Investigating a flaky test | Agentic |
| Visual or copy changes across many pages | Deterministic, with visual testing tools |
| Validating that a goal is still reachable after a redesign | Agentic |
The most valuable pattern is the handover. Use an agent to explore or reproduce, then capture what it learned as a deterministic test that runs cheaply forever. We discussed the same pattern in Playwright Test Agents and the truth about self-healing tests.
Pros and cons
Agentic E2E testing
Pros: adapts to UI variation, verifies goals rather than scripts, good for exploration and bug reproduction, high reliability with a well-built execution layer.
Cons: $15 to $30 per run in Slack’s setup, minutes per run, non-deterministic paths, sensitive to environment stability.
Deterministic E2E testing
Pros: fast, cheap, repeatable, easy to parallelise, clear failure signals.
Cons: brittle to UI change, enforces one path, needs maintenance, and AI-generated versions need good specifications to handle complex flows.
What this means for your team
Slack’s study is the best public evidence so far that agentic and deterministic E2E tests are complements, not competitors. Keep deterministic tests as the foundation of CI. Add agentic testing for exploration, debugging and reproduction, where adaptability is worth the cost.
When you evaluate tools, measure total cost per successful task and classify failures honestly. And invest in the execution environment. A stable login and clean test data may improve your agentic results more than switching models.
Frequently asked questions
What did Slack find about agentic E2E testing?
Across 200+ runs, agents using Playwright MCP had 0 to 12% failure rates, Playwright CLI agents 12 to 20%, and generated Playwright tests about 8% on a simple flow and 48% on a complex one. Agent runs cost about $15 to $30 each.
Is Playwright MCP better than Playwright CLI for testing?
In Slack’s experiment, MCP was more reliable, faster and used fewer total tokens, mainly because it needed fewer turns. Results depend on your setup, so measure on your own flows.
Should agentic tests run in CI?
Slack suggests that, because of cost, agent-driven testing is currently better suited to targeted debugging and exploration than to high-frequency CI execution. Deterministic tests remain the CI foundation.
Why are agentic tests expensive?
Each turn re-sends the system prompt and conversation history, so cost grows with turn count and context size. Slack found most tokens were retransmission of previously seen content.
Where does agentic testing fit in the testing pyramid?
Slack adds it as a new top layer above E2E, used for exploring complex behaviour, debugging flaky workflows and reproducing production bugs.
Did natural language or YAML work better as agent input?
Slack tested both formats but its published summary reports results by execution model rather than by input type, so the post does not declare a winner. It frames the difference as how detail is represented, with YAML making the mapping from steps to actions more explicit.