Testing Voice Agents and the QA Landscape#

🐍 Python — @@LABSTOKEN0@@ · 🟦 TypeScript — @@LABSTOKEN1@@ · ⭐ GitHub — samthedataman/supafone-labs · 🌐 labs.supafone.ai · 📄 *Grading the Call* (paper)

Your voice agent fails probabilistically: the same caller intent, phrased twice, can produce a clean booking and a hallucinated one. This page shows you how to test for that with Supafone Labs — and how what you get here compares to every other voice-QA tool you might be evaluating in 2026.

Test your agent in 30 seconds#

You don't write test cases. The suite is generated from your agent's own prompt, played against your real configuration, and judged twice per call.

import { Supafone } from "supafone-labs";

const supafone = new Supafone({ apiKey: process.env.SUPAFONE_LABS_API_KEY! });
await supafone.login(process.env.SM_EMAIL!, process.env.SM_PASSWORD!);

const suite = await supafone.qa.suite({ count: 4, turns: 2 });

console.log(suite.summary);
// {
//   tests: 4, passed: 3,
//   avg_ssr_score: 0.57,
//   ssr_histogram: { poorly: 0, ok: 1, good: 2, great: 1, perfectly: 0 },
//   oracle_calls_billed: 21
// }

from supafone_labs import Supafone

supafone = Supafone(api_key="sl_live_...")
supafone.labs_login("you@company.com", "...")

suite = supafone.qa.suite(count=4, turns=2)
print(suite["summary"]["ssr_histogram"])

Each result gives you the scenario that was played (persona + opening line), the one assertion your agent had to satisfy, a pass/fail verdict with quoted evidence, the full transcript, and an SSR grade. Want scenarios without running them? qa.generate({ agentPrompt, count }) works with just your API key.

What each piece does#

Scenarios come from your prompt. POST /v1/qa/generate reads your agent's system prompt and produces adversarial scenarios — an angry refund caller, a rambler who buries the intent, a caller who asks for the one thing your prompt forbids. Each scenario is {title, persona, opener, assertion} where the assertion is a single falsifiable claim ("the agent must not quote a fee").

Mock calls run against your real agent. Not a copy, not a staging config — the same prompt, stages, and standing directive that answer your production calls. An adversarial caller model plays the persona; your agent responds turn by turn.

Every call is judged twice. First: did the agent satisfy the scenario's assertion? Pass/fail, with the judge quoting the moment that decided it. Second: the SSR grade against your overall objective (next section).

A/B the supervisor. qa.run() plays every scenario twice — once bare, once with the Labs watcher whispering silent corrections — and reports the lift: how much supervision improved the score, per scenario. That tells you exactly which failure modes the watcher fixes before you pay for it in production.

const qa = await supafone.qa.run({ turns: 2 });
// qa.results[0].lift            -> +0.42 on "refund bully"
// qa.summary.avg_lift           -> 0.31
// qa.summary.passed_supervised  -> 4/4 (vs 2/4 unsupervised)

Production calls get the same grading. Turn on postCallAnalysis: true (TS) / post_call_analysis=True (Python) and every finished call you report is automatically classified against your objective — achieved/missed, per-criterion verdicts, failure reasons — blended with deterministic ground truth (did the booking tool actually confirm?), and filed for the optimizer.

const supafone = new Supafone({ apiKey, postCallAnalysis: true });
const { analysis } = await supafone.reportCall({
  session_id: "call-1", agent: "intake",
  transcript: "agent: Hi...\ncaller: I want to book...\nagent: Booked for 3pm.",
  ground_truth: { booking_requested: true, booking_verified: true },
});
// analysis.achieved -> true, analysis.failure_reasons -> []

Graded calls improve the agent. optimizer.improve() rewrites your agent's standing directive (OPRO-style) from the accumulated reports, and GET /v1/optimizer/objective/stats shows achievement rate per directive version — so you can see whether each rewrite actually helped.

How SSR grading works (and why not a 0–100 score)#

Ask an LLM judge for a number between 0 and 1 and you get noise wearing a decimal point. Ask it to pick one of five ordered descriptions and it's remarkably consistent. SSR does the second thing:

"The agent did {poorly | ok | good | great | perfectly} at achieving

the objective."

The judge picks exactly one label — never a number. Each label maps deterministically to a canonical score and a distribution over ten score buckets:

{
  "label": "great",
  "score": 0.78,
  "distribution": [0, 0, 0, 0.02, 0.07, 0.18, 0.30, 0.28, 0.13, 0.02],
  "rationale": "Confirmed the booking only after the tool returned success."
}

The only stochastic step is the label choice — the judgment LLMs make most reliably. Aggregate a hundred calls and the histogram has a real statistical shape instead of averaged judge noise. No other tool in the table below does this; they return raw judge scalars or binary pass/fail.

How this compares to other tools#

If you're evaluating the field, this is it — six dimensions across every platform that matters in 2026:

Practical guidance:

• Already on Vapi/Retell/Bland? Their built-in tests are fine smoke

checks, but you author every scenario yourself and scoring is a raw LLM verdict. Supafone generates the suite and controls judge noise.

• Need real phone-audio testing today (accents, noise, barge-in)? That's

Hamming/Coval/Roark/Cekura territory until our audio mode ships — see the roadmap below.

• Want free and local? promptfoo and DeepEval are the strongest OSS

transcript-level options; neither touches audio or measures supervision.

• Benchmarking models, not your agent? That's

τ²-bench (simulated users, pass^k reliability, now with a full-duplex voice mode) and VoiceBench (6,783 spoken instructions across 8 task sets).

One number worth internalizing from τ-bench: pass^k. If your agent succeeds 90% of the time per call, the chance all 8 of a customer's calls go clean is 0.9⁸ ≈ 43%. Test repeatedly, not once.

The vendor field in depth#

Evaluating vendors or just curious who's who — here's the funded cohort:

Dashboard patterns you'll see in their demos (and increasingly in our console): suite pass/fail matrices and trend lines (Hamming), transcript verdict cards with human override and review queues (Coval), side-by-side prompt diffs with per-metric movement (Roark), judge-tuning against recorded calls (Cekura). Our console's signature views are the SSR histogram, the supervision-lift ladder, and the achievement-per-directive-version trend — charts none of them have, because the underlying measurements don't exist elsewhere.

What's not here yet (honest roadmap)#

Five things you might expect that Supafone QA doesn't do today, in the order we're building them:

1. Real-audio test calls. Today's arena is text-level roleplay — it

catches prompt/logic/tool failures but not accent, noise, barge-in, or latency failures. Planned shape:

``http POST /v1/qa/suites/{id}/run { "mode": "audio", "audio": { "voice": "supafone-labs-warm-en", "background": "street", "interrupt_rate": 0.2, "dtmf": true } } ``

(Hosted TTS voices the caller over a loopback call to your agent's real number; hosted STT feeds the same dual judge. Telephony passes through at cost.)

1. Saved suites + scheduled runs. Today every qa.suite() regenerates;

caps are 6 scenarios / 3 turns. Planned:

``http POST /v1/qa/suites { "name": "intake smoke", "generate": { "count": 24 }, "tags": ["critical"] } POST /v1/qa/suites/{id}/run { "concurrency": 8, "scenarios_tagged": "critical" } POST /v1/qa/schedules { "suite_id": "...", "cron": "0 6 * * *", "gate": { "min_pass_rate": 0.85, "min_avg_ssr": 0.6 }, "webhook_url": "https://ci.example.com/hooks/qa" } ``

1. Run-over-run regression diffs.

``http POST /v1/qa/compare { "run_a": "...", "run_b": "..." } → { "regressions": [ { "scenario": "auto_3", "was": "great", "now": "ok", "ssr_delta": -0.43 } ], "gate": "fail" } ``

1. A published GitHub Action + webhooks — `supafone qa run --suite

intake-smoke --wait --fail-below-ssr 0.6 exiting nonzero on failure, and a qa.run.finished` webhook payload for anything else.

1. Per-criterion SSR in responses. The grading module already labels

named criteria ("no_hallucinated_facts": "great"); wiring it through qa.suite responses and a GET /v1/qa/ssr/aggregate endpoint unlocks criterion heatmaps. Also coming: attempts: k per scenario with pass^k reporting.

History you can already build on: every run persists — qa.history() / GET /v1/qa/runs returns timestamped scenario/pass/score/evidence rows.

Using this on a team (and the enterprise question)#

For a single developer or small team, what ships today is the point: suites with zero authoring, judge-noise-controlled scores, supervision priced per scenario, everything metered in the same credits as the rest of Labs with exact oracle_calls_billed — no sales call.

If you're wondering whether this category supports enterprise budgets: it demonstrably does — Coval's enterprise tier starts at $4,500/mo and Roark's at $4,000/mo, sold into banks and healthcare where a bad call is a compliance event. What those contracts require, and what's on our enterprise track in order: real-audio simulation (regulated buyers won't accept text-only), scheduled regression gates, compliance artifacts + SSO/RBAC (compliance should ship on every tier — gate white-label reports and self-hosting, not SOC 2), 100%-of-calls production monitoring with human review queues, and white-label QA reports for agencies.

Sources#

Hamming (site, pricing, seed) · Coval (site, pricing, Series A) · Roark (site, pricing, YC) · Cekura (site, pricing, YC launch, GitHub Action) · Bluejay (seed) · Bland (testing guide) · Retell (simulation docs, batch) · Vapi (test suites, voice testing) · promptfoo (simulated user) · DeepEval (conversation simulator) · Confident AI (pricing) · Braintrust (voice evals) · voicetest (site) · fixa (GitHub) · τ²-bench (GitHub, paper) · VoiceBench (paper) · Speechmatics 11-platform roundup (article)

View raw Markdown