Supafone Supervisor#

Supafone Supervisor runs beside an agent, off the realtime hot path. It observes the call, returns a silent directive only when the live agent needs help, and can score and QA the call after it ends.

For the practical developer view—multilingual continuity, provider switching, cross-worker state, safety, telemetry, and failover—start with Production Voice AI: Daily Problems Supafone Solves.

The first principle: the model speaking cannot fully supervise itself#

Realtime voice models are optimized to answer quickly. A production supervisor has a different job: watch patterns across turns, compare spoken claims with tool ground truth, remember the operator's objective, notice a change in language or urgency, and decide whether an intervention is worth interrupting the agent's current trajectory.

Putting both jobs in one prompt creates a structural conflict. More reasoning adds latency; less reasoning misses the moment. Supafone separates the roles:

text

speaking model                         supervisor model
--------------                        ----------------
fast, natural response                slower cross-turn reasoning
owns the customer audio               never speaks to the customer
uses tools and follows stages         checks tool truth and stage progress
continues if supervisor is absent     emits a bounded silent directive or no-op

The supervisor is not a replacement agent and not a transcript summarizer. It is a second control loop beside the call.

The secret sauce: empathy as observable patterns#

“Empathy” is not a personality adjective in the runtime. It is a changing set of observable patterns that affect what the agent should do next:

intent: what outcome the caller is actually trying to reach,
urgency: whether waiting, escalation, or a shorter path matters,
emotion: confusion, frustration, fear, confidence, or relief across turns,
language: an explicit request or clear utterance in an approved language,
trust: whether the agent acknowledged, verified, and followed through,
progress: whether the current workflow stage is advancing or looping,
truth: whether a booking, transfer, send, or CRM action really succeeded.

The Supervisor maintains that belief state over time. It does not route from a name, accent, nationality, or presumed demographic. It waits for evidence, compares the call with the operator's objective and tool results, and whispers only when a short directive is likely to improve the outcome.

The supervisor loop#

text

provider event
    -> normalize into one call contract
    -> update intent / emotion / language / stage / tool truth
    -> compare with objective, policy, and standing directive
    -> guard on evidence, tenant, provider, cooldown, and timeout
    -> compile one silent native instruction—or do nothing
    -> observe the next turn and verify whether it helped
    -> grade the completed call and improve the standing directive

This is why the framework can become more useful without taking over the live audio path. The speaking agent stays fast; the supervisor accumulates context, detects patterns, and closes the verification loop.

Model agnostic by construction#

The contract is between call events and supervisor directives, not between Supafone and one model vendor. The speaking model, supervisor model, carrier, STT, and TTS can be selected independently when the provider exposes the required control surface.

Adapters translate provider-native events into the canonical state and compile the resulting directive back to the provider's native silent channel. A team can therefore keep Vapi, Retell, Ultravox, OpenAI Realtime, LiveKit, Pipecat, Deepgram, ElevenLabs, or another compatible stack while retaining the same supervision, QA, telemetry, and improvement loop. See Framework Support for exact capabilities and caveats.

The managed Ultravox-compatible Agent Factory is a compatibility lane for provisioning a complete hosted agent with supervision already attached. Native realtime Agent Factory is a separate first-class path for swapping among the four native S2S provider families while keeping the carrier contract. Both Agent Factory paths support coaching when enabled and configured. The Supervisor contract also works when Supafone did not create the agent.

Supervisor coaching across all five speaking families#

Enable supervisor: true (Python "supervisor": True) when creating an agent, or supply the existing managed/BYOK supervisor configuration. Supervisor is independent of the speaking model: Ultravox, OpenAI, Gemini, Grok, and Hydra all support hosted coaching. The same call-scoped coach is used for native browser sessions and Supafone-managed, Twilio, Telnyx, Plivo, and SIP calls. Each transport still needs its own configuration and live validation.

import { Supafone, HydraS2S } from "supafone-labs";

const client = new Supafone({ apiKey: process.env.SUPAFONE_TOKEN! });
const engine = new HydraS2S(client, { model: "hydra-v1.1", voice: "maya" });
await engine.create({
  agentKey: "coached-intake",
  name: "Coached intake",
  description: "Understand the caller's request and verify the booking result.",
  supervisor: true,
});

python

from supafone_labs import Supafone, OpenAIS2S

client = Supafone(api_key="sl_live_...")
engine = OpenAIS2S(client, model="gpt-realtime-2.1", voice="marin")
engine.create({
    "agentKey": "coached-intake",
    "name": "Coached intake",
    "description": "Understand the request and verify the booking result.",
    "supervisor": True,
})

Observe, reason, return guidance#

Speaking path	Observation and delivery
Managed Ultravox, including compatible custom TTS	Existing call observation and deferred guidance delivery
Native OpenAI, Gemini, Grok	Provider transcripts and server tool outcomes inform a background coach; the model receives available guidance in a `check_guidance` tool result
Native Hydra	No native transcripts. `check_guidance(context)` supplies model-reported context; the coach checks it alongside server tool outcomes and returns guidance through the same tool result

The native model is instructed to check for guidance at appropriate turn/tool boundaries. Supervisor never speaks to the caller, forces a new response, or interrupts the audio stream. A slow or unavailable Supervisor yields no instruction and the call continues. Tool-result delivery is a request by the speaking model, not a guarantee that it checks on every turn or follows a note.

Hydra context is explicitly recorded as model_reported_context, not a transcript or an independently verified quote. Provider transcript paths use provider_transcript. The server's actual tool result remains authoritative for whether a booking or another operation succeeded. Hydra's persona and voice remain fixed during the session; a coaching tool result does not rewrite them.

Supported, enabled, and ready are different#

Native runtime metadata separates supervisor: true (supported) from supervisor_enabled (the saved agent setting), supervisor_credentials_configured (whether the Supervisor credential resolves), supervisor_observation (provider_transcript or model_reported_context), and supervisor_delivery: "tool_result". Support or an enabled setting does not establish working credentials or a live coach. Even configured credentials do not verify provider access or successful inference. Configure the selected managed or BYOK Supervisor model separately from the speaking model, and inspect call activity before treating coaching as running.

See Supervisor Models: Managed and BYOK for model configuration. The standalone adapter matrix describes external sessions; Hydra's hosted coaching does not imply a standalone Hydra Supervisor adapter exists.

Enable supervision#

The SDK enables supervision in hosted agent creation by default. The selected Supervisor model still needs configured managed or BYOK credentials. Native coaching is available across the five speaking families; recording, QA artifacts, and transcript availability remain separate runtime capabilities.

Python:

python

from supafone_labs import Supafone

supafone = Supafone(api_key="sl_live_...")

TypeScript:

import { Supafone } from "supafone-labs";

const supafone = new Supafone({ apiKey: process.env.SUPAFONE_TOKEN! });

Use the optional supervisor boolean to disable or re-enable this default. Previous constructor options remain accepted as deprecated compatibility aliases, so existing integrations continue to run unchanged.

What Supafone monitors#

caller intent, urgency, language, and emotion,
transcript contradictions,
tool result failures,
unverified booking, sending, pricing, or policy claims,
compliance rules such as no fee quotes or no legal/medical advice,
whether the agent is following the current standing directive.

Transcript Source and STT Model#

Supafone Supervisor does not force every provider through one speech-to-text model. It selects exactly one transcript source for each call:

Call path	Transcript source	Default model
Provider emits usable transcript events	The provider's transcript stream	The provider controls its STT model
Supafone Labs multilingual audio tap	Deepgram streaming STT	`nova-3`, `language=multi`
Host-integrated narrowband phone tap	The host's configured Deepgram consumer	Host controlled; Supafone's current Twilio reference defaults to `nova-2-phonecall`

The narrowband phone default is deliberate: Twilio PSTN audio arrives as 8 kHz mu-law. The multilingual SDK tap uses Nova-3 when language tagging and live code-switching are required. Set DEEPGRAM_MODEL in a host deployment to change its telephony-tap model. Do not run the Deepgram tap when the selected agent provider already supplies the required transcript and language metadata; that would duplicate turns and transcription cost.

Enable on Hosted Agents#

json

{
  "labs": {
    "enabled": true,
    "model": "gemma"
  }
}

Equivalent legacy fields:

json

{
  "voice_watcher": true,
  "voice_watcher_model": "gemma"
}

Bring-Your-Stack Supervision#

python

from supafone_labs import SupafoneLabs

brain = SupafoneLabs(
    provider="vapi",
    llm="hosted",
    agent_label="intake",
)

result = await brain.observe(raw_event)

for action in result.actions:
    await deliver_to_voice_platform(action)

Two ways the whisper lands#

Every directive reaches the live agent through one of two silent-injection modes, picked by what the framework exposes:

Mode A — native silent event. Speech-to-speech models take a vendor event

that adds context without triggering speech (Ultravox send_data_message/inject_message, OpenAI Realtime conversation.item.create with no response.create, ElevenLabs contextual_update, Gemini Live clientContent).

Mode B — own the LLM. For STT→LLM→TTS pipelines Supafone plugs in as the

LLM and splices a system/developer message into the prompt (Retell and LiveKit custom-LLM loops; Vapi and Deepgram support both modes).

The release gate covers fourteen runtimes. Twelve use native control or developer-owned context, Bland remains observation-only, and Cartesia Line requires an explicit host hook. Pipecat is a first-class developer-owned context integration. The exact primitive and acceptance criterion live in Framework coverage.

Outcome Loop#

Log the finished call:

await supafone.reportCall({
  session_id: "call-123",
  agent: "intake",
  score: 0.82,
  outcome: "clean",
  summary: "Caller scheduled a follow-up without unsupported claims.",
  nudges: 2,
  turns: 14,
  language: "en"
});

Or classify a transcript against an objective:

bash

curl https://api.labs.supafone.ai/v1/calls/classify \
  -X POST \
  -H "Authorization: Bearer $SUPAFONE_LABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "call-123",
    "agent": "intake",
    "transcript": "caller: what do you charge?\nagent: I cannot quote fees here.",
    "nudges": 1
  }'

Improve the standing directive:

const improved = await supafone.optimizer.improve("intake");
console.log(improved.version, improved.text);

Read it:

bash

curl "https://api.labs.supafone.ai/v1/optimizer/standing?agent=intake" \
  -H "Authorization: Bearer $SUPAFONE_LABS_API_KEY"

Degrade Safety#

The supervisor is timeout-bounded and off the hot path. If the oracle fails, times out, hits a balance or cap error, or decides no intervention is needed, it returns no directive and the call continues normally.

View raw Markdown