Roark

Visit Website

Introduction

Roark is a testing and evaluation platform for voice AI, explicitly not another agent builder. It works with agents you already run on stacks such as Vapi, Retell, LiveKit and Pipecat, plus your own.

Before launch, it runs your agent against hundreds of simulated callers built from your real call types: the angry caller, the rambler, the interrupter. Adversarial callers try prompt injection, jailbreaks and social engineering to check the agent holds policy. Simulations include native accents, code-switching and background noise for each market, and load tests with peak-volume concurrency plus always-on health checks.

Regression testing is a first-class idea. You rerun the whole suite on every change and diff it against the last green baseline, so fixing one caller does not break another, and prompt or model changes can run the suite before they merge as quality gates for conversations.

In production, every call is scored as it lands. Issues are filed, alerts fire, and dashboards and OpenTelemetry traces are available for voice calls and chat threads. Where many tools grade a transcript with an LLM, Roark runs purpose-built audio models on the call itself to measure what the customer actually heard, across 500+ audio-native metrics such as pronunciation, empathy and resolution. Human review lets you label calls and align metrics with ground truth.

Roark also publishes a public leaderboard of realtime voice stacks, calling every speech-to-text, language model and speech engine combination over the real phone network and scoring them with the same engine.

Voice agents fail in ways text agents do not: interruptions, background noise, accents, long pauses and people changing their minds mid-sentence. Testing them properly means simulating those conditions repeatedly, not just reading transcripts after the fact.

When trialling Roark, replay a handful of real calls that went badly and check whether its simulations and evaluations would have flagged them. Then add those scenarios to a regression suite that runs whenever prompts, models or call flows change.

Measure the right things as well: task completion, correct handoffs to humans, latency between turns and how often the agent talks over the caller. Those metrics map directly to customer experience and make regressions easy to explain to non-engineers.

Finally, keep a human in the loop for new call flows. Listen to a sample of simulated calls each week, because automated scores can miss tone and awkward phrasing that real callers notice immediately.

Launch and background: Y Combinator’s company directory lists Roark in its Winter 2025 batch, with the one-liner “Test, monitor, and improve your voice agents”. Roark was launched on Hacker News on 17 February 2025 (“Launch HN: Roark (YC W25)”) and has a Product Hunt listing.

Features

1. Caller simulation: Hundreds of personas built from real call types.
2. Adversarial callers: Prompt injection, jailbreaks and social engineering.
3. Accents and noise: Code-switching and background noise per market.
4. Load and health checks: Peak concurrency and always-on monitoring.
5. Baseline diffs: Compare each run against the last green suite.
6. Merge gates: Run the suite before prompt or model changes merge.
7. Audio-native scoring: 500+ metrics measured on audio, not just transcripts.
8. Human review: Label calls and set ground truth.
9. OTEL traces: Traces for voice and chat.
10. Voice model leaderboard: Public benchmarks of voice stacks.

Pros

1. Audio-level analysis: Catches problems a transcript hides.
2. Plain-dollar pricing: No seat licences.

Cons

1. Voice-first: Chat support exists, but voice is the core.
2. Provider costs extra: Simulation minutes add telephony and model costs.
3. Enterprise minimum: Large plans start at $4,000 a month.

Usage and Links (Example)

1. Sign up with a work email for $50 of credit.
2. Connect your agent from Vapi, Retell, LiveKit or Pipecat.
3. Generate simulated callers from your call types and run a baseline suite.
4. Turn on production call scoring and route alerts to Slack.

Resources:

• Roark website

Pricing

Roark is usage-based: pay as you go from $0 with $50 free credit for a work email, Team at $500 a month spent as usage, and Enterprise from $4,000 a month committed. Simulation is billed per minute (for example $0.15 a minute on pay as you go) plus provider costs.

Recommended for?

1. Teams running voice agents in production in support, healthcare, finance or hospitality.
2. Agencies validating client agents before go-live.

Conclusion

Roark’s audio-native scoring and baseline diffing make it one of the most thorough voice-agent QA platforms.

Start with pay-as-you-go credit and a small persona set before scaling simulations.

On CheckOps, Roark is filed under the Release & CI gates and Production monitoring & observability stages. Related tools listed here include Hamming, Coval, Cekura, LangSmith, Langfuse and Promptfoo. Before standardising on one, compare recent maintenance activity, the pricing model as your usage grows, and the integrations you rely on today. Information on this page was verified against the official website, documentation and repository on 11 November 2026.