Native realtime Agent Factory#

Supafone's S2S harness connects a provider-native speech-to-speech model to one agent's prompt, supported tools, fixed stages, and browser or phone audio. Agent Factory creates that durable agent; its realtime selection chooses the speaking model. Updating the selection reuses the native agent contract for the next session. Model voices and behavior remain provider-specific.

The native transport is separate from the managed Ultravox compatibility path and from Supafone Supervisor, which observes and coaches existing supported agent stacks. Omitting realtime when creating an agent keeps the managed compatibility runtime. Apply UltravoxS2S (or update realtime: null) to switch an existing native agent back to that runtime.

Use this guide when the speaking model itself should own the live audio loop. Use Supafone Supervisor when you want to keep another agent stack and add supervision.

One interface, five provider families#

The public SupafoneS2S superclass has UltravoxS2S, OpenAIS2S, GeminiS2S, GrokS2S, and HydraS2S provider classes. They produce the same Agent Factory selection shape, so an application can choose a provider and reuse the same create, update, and preview workflow. See Shared S2S Interface for Python and TypeScript examples.

Ultravox remains the default managed runtime. The other four families provide six native model choices below. A common interface does not erase differences in transcripts, voices, tool behavior, or runtime features.

Supported catalog#

The catalog is the source of truth. Call GET /api/v1/agents/catalog or GET /api/v1/labs/capabilities instead of hard-coding the list in a builder.

GPT Live delegates reasoning and tools to GPT-5.6 Luna. The OpenAI key used by the workspace must be allowed to use both models. Gemini is marked preview because the provider API is still subject to change.

Hydra currently supports English only. It uses native model voices and does not emit transcript events. Its initial instructions and voice are fixed for a session. See Hydra capabilities before choosing it for a workflow that requires transcripts or live persona changes.

Every catalog entry currently advertises both authenticated browser and phone transport for these phone providers:

• Supafone-managed phone (native / supafone_managed)
• BYO Twilio (byo_twilio)
• BYO Telnyx (byo_telnyx)
• BYO Plivo (byo_plivo)
• BYO SIP (byo_sip)

A catalog entry means an adapter exists. The selected model needs a configured Supafone platform key or an account BYOK key. Phone delivery also needs a reachable public application URL and managed or BYO carrier setup. Catalog presence alone does not verify model access or a successful live call.

Create a native realtime agent#

The realtime object contains only the provider, model, and voice. It never contains a browser-visible credential in the returned agent document.

TypeScript:

import { Supafone } from "supafone-labs";

const supafone = new Supafone({
  apiKey: process.env.SUPAFONE_API_KEY!,
});

const agent = await supafone.labs.agents.createInbound({
  agentKey: "northline-realtime",
  name: "Northline realtime intake",
  assistantName: "Maya",
  description: "Answer questions, capture the request, and book the next step.",
  realtime: {
    provider: "google",
    model: "gemini-3.1-flash-live-preview",
    voice: "Puck",
  },
  telephony: {
    mode: "supafone_managed",
    provider: "supafone",
  },
});

Python:

from supafone_labs import Supafone

supafone = Supafone(api_key="sl_live_...")
agent = supafone.labs.agents.create_outbound({
    "agentKey": "northline-realtime-follow-up",
    "name": "Northline realtime follow-up",
    "goal": "Call a consented lead and book a consultation.",
    "realtime": {
        "provider": "xai",
        "model": "grok-voice-latest",
        "voice": "eve",
    },
    "telephony": {"mode": "byok", "provider": "telnyx"},
})

Native realtime agents use a fixed three-stage contract: intake, booking, and confirmation. The hosted call planner is skipped because provider-native audio sessions cannot safely accept arbitrary client-generated stage payloads. The selected model, voice, and tools remain account-scoped and durable.

Managed credentials and optional BYOK#

Use your Supafone application key to create agents. For each selected model, the server uses the account's encrypted provider key when present; otherwise it uses Supafone's configured platform key. Customers do not need their own provider account when that platform key is available. Keys never belong in the agent's realtime selection or browser code.

Check GET /api/v1/labs/runtime?provider=google (or openai, xai, smallest). Status reports configured, connected, and source: account for BYOK, platform for managed credentials, none for missing configuration, and invalid for an unreadable saved key. connected means a credential resolves, not that a provider has accepted a live connection. Verify model access with a preview.

To override the platform key with your own provider account, configure BYOK:

curl "$SUPAFONE_API_BASE_URL/api/v1/labs/runtime" \
  -X PUT \
  -H "Authorization: Bearer $SUPAFONE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "google",
    "credentials": {"api_key": "AIza..."}
  }'

Use openai, google, xai, or smallest for native realtime. Google accepts either a Gemini or Google API key. Environment fallback names are OPENAI_API_KEY, GEMINI_API_KEY/GOOGLE_API_KEY, XAI_API_KEY, and SMALLEST_API_KEY when the platform is configured to supply them. Do not put a key inside realtime or log it in a client payload.

Optional BYOK configuration is also available through both SDKs:

await supafone.labs.runtime.configure({
  provider: "openai",
  credentials: { apiKey: process.env.OPENAI_API_KEY! },
});

import os

supafone.labs.runtime.configure({
    "provider": "xai",
    "credentials": {"api_key": os.environ["XAI_API_KEY"]},
})

GET /api/v1/labs/runtime?provider=google returns {configured, connected, source} without returning the key. A selected key must match the selected realtime provider; cross-provider or conflicting keys are rejected.

Browser preview#

testCall/test_call starts an authenticated, rate-limited browser session. The response contains a one-use WebSocket ticket and the provider's audio sample rates. It is a browser preview, not a PSTN call and not a promise that carrier routing is ready.

const preview = await supafone.labs.agents.testCall("northline-realtime");
console.log(preview.browser_session.transport); // "supafone_realtime"
console.log(preview.browser_session.websocket_url);

preview = supafone.labs.agents.test_call("northline-realtime")
print(preview["browser_session"]["transport"])

The browser connects to wss://YOUR_API_HOST/api/v1/realtime/connect with the short-lived ticket. The server owns the provider WebSocket and tool execution; model credentials never cross into the browser.

Phone calls and carrier selection#

Use the default Supafone-managed telephony path when the workspace has an approved managed number. Use BYO telephony when the customer owns the carrier account:

await supafone.labs.telephony.configure({
  mode: "byok",
  provider: "twilio", // "telnyx", "plivo", or "sip"
  credentials: {
    // Supply the required fields for the selected carrier on your server.
  },
});

This configures the account's phone provider and can affect agents using that account's carrier settings. It is not a per-agent model update. Check existing number assignments and routing before changing a shared carrier configuration.

The carrier audio adapter converts media to mono PCM16 and returns model audio at the catalog's output rate. For outbound calls, verify the caller ID, carrier credentials, public HTTPS/WebSocket URL, destination policy, and account credit before dialing. For inbound calls, carrier application configuration is a separate admin step:

• Telnyx: PUT /api/v1/realtime/phone/inbound/{agent_id}/byo_telnyx/config,

then point the Telnyx application at the returned webhook URL. Telnyx Ed25519 webhook signatures are required.

• Plivo: PUT /api/v1/realtime/phone/inbound/{agent_id}/byo_plivo/config,

then configure the returned answer, hangup, and stream-status URLs. Plivo signature v3 verification is required.

• SIP: configure LiveKit SIP, then call

POST /api/v1/realtime/sip/agents/{agent_id}/inbound with the number and allowed source addresses. Send signed LiveKit events to /api/v1/realtime/sip/events.

Twilio inbound and outbound use the existing managed/BYOK telephony setup and its carrier webhook checks. See Phone Numbers and the private product's SIP architecture guide for carrier-specific infrastructure.

REST contract#

Example create payload:

{
  "agent_key": "northline-realtime",
  "name": "Northline realtime intake",
  "direction": "inbound",
  "realtime": {
    "provider": "openai",
    "model": "gpt-realtime-2.1",
    "voice": "marin"
  },
  "telephony": {
    "mode": "supafone_managed",
    "provider": "supafone"
  }
}

Hydra capabilities#

Smallest AI's Hydra models provide direct audio and tool calls, with no native transcript events. Supafone therefore cannot show or grade a Hydra transcript as though one had been received. A separate transcription system is not implicitly enabled. See Smallest AI's overview.

Hydra accepts mono PCM16 at 16 kHz. Version hydra-v1.0 returns 48 kHz audio; hydra-v1.1 returns 24 kHz. The browser or carrier adapter uses the selected model's sample rates. Use a voice from that version's catalog rather than carrying a voice ID from another provider. See the Hydra model card.

The opening instructions and voice are fixed when Hydra connects; changing those requires a new session. Only tool definitions can be updated mid-session. Supafone keeps stage authority on its server and communicates approved stage instructions through tool results. This is not a live rewrite of Hydra's persona. See session behavior and tool calling.

Configure SMALLEST_API_KEY on the Supafone server for managed Hydra access, or store an optional account key with provider smallest. Missing credentials remain setup required. Adding the adapter does not configure a production key or establish successful live calls.

Feature boundaries#

Native realtime is intentionally explicit. Current runtime responses report:

• browser preview and carrier phone transport: supported;
• fixed intake → booking → confirmation stages: supported;
• provider-native tools and account-scoped agent configuration: supported;
• transcripts: provider-dependent; Hydra has no native transcript events;
• recording, Supafone Supervisor coaching, human transfer, DTMF navigation, and

specialist-team handoff, public web widgets, and live language/voice profile switching: not implemented on this transport;

• live carrier quality, regional reachability, and model access: require a

deployment test with real credentials.

When recording, Supervisor, transfer, or a full planner is required, use the managed Ultravox path or bring an existing compatible stack to Supafone Supervisor instead.

Troubleshooting checklist#

1. invalid_credentials: reconnect the key for the selected provider and check

the workspace's provider status.

2. setup_required: configure carrier credentials, caller ID, and a public

HTTPS/WebSocket BASE_URL; SIP also requires LiveKit settings.

3. Browser session unavailable: confirm the agent is active and use the

authenticated testCall route rather than connecting to the provider API from the browser.

4. Inbound carrier rejection: verify the returned webhook URL, signature keys,

allowed number list, and that the agent still has the selected realtime model.

5. SIP failures: configure LIVEKIT_URL, LIVEKIT_API_KEY,

LIVEKIT_API_SECRET, and LIVEKIT_SIP_URI, then verify the signed webhook reaches /api/v1/realtime/sip/events.

Automated tests cover catalog validation, provider adapters, browser tickets, carrier media, Telnyx/Plivo signature checks, SIP lifecycle, duplicate webhook handling, and PostgreSQL claim/finalization. They do not place live calls.

View raw Markdown