> ## Documentation Index
> Fetch the complete documentation index at: https://developers.telnyx.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime voice and avatar conversations over WebRTC

> Connect a browser to a Telnyx AI Assistant over WebRTC: send microphone audio and camera video as media tracks, play the assistant's voice and avatar video, and exchange conversation events on a data channel.

**Beta:** Realtime conversations over WebRTC are in beta.

Connect a browser or other WebRTC client directly to a [Telnyx AI Assistant](https://portal.telnyx.com/#/ai/assistants). Media travels as WebRTC tracks: the caller's microphone (and optionally camera) to Telnyx, the assistant's voice (and optionally its [avatar](https://developers.telnyx.com/docs/inference/ai-assistants/avatars.md) video) back. Conversation events travel as JSON on a WebRTC data channel named `oai-events`.

The assistant's behavior — instructions, model, voice, tools, transcription and interruption settings — is configured on the assistant itself. The WebRTC connection carries media and events only.

Use WebRTC for browser and mobile clients. For a server-side integration that streams PCM audio, use the [conversation WebSocket](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md) instead.

## How a call is set up

```
Browser                         Application server                  Telnyx
   │ 1. create offer (mic, data channel,                               │
   │    optional video), gather ICE                                    │
   │── 2. SDP offer ──────────────────▶│                               │
   │                                   │── 3. POST /calls (API key) ──▶│
   │                                   │◀─ 4. 201 SDP answer ──────────│
   │◀─ 5. SDP answer ──────────────────│                               │
   │══ 6. media + oai-events data channel, relayed through Telnyx TURN ═│
```

1. The browser creates an `RTCPeerConnection` that relays through Telnyx TURN, adds the microphone track, the `oai-events` data channel and, for an avatar or camera, a video transceiver. It waits for ICE gathering to complete.
2. The browser sends the SDP offer to the application server.
3. The application server forwards the offer to `POST /v2/ai/assistants/{assistant_id}/calls` with its Telnyx API key. The API key never reaches the browser.
4. Telnyx starts a conversation with the assistant and returns the SDP answer.
5. The application server returns the answer to the browser, which applies it.
6. Media and the data channel connect. The conversation, including the assistant's greeting, starts once the `oai-events` data channel opens.

### Migrate an existing avatar client

Replace the avatar WebSocket connection and its signaling frames with the call setup above. Do not send `avatar.session.ready`, `avatar.webrtc.offer`, `avatar.ice_candidate`, `avatar.webrtc.answer`, or `avatar.media.ready` over the data channel. Mid-call renegotiation uses the separate `webrtc.offer` and `webrtc.answer` events described below.

The greeting starts when the data channel opens, rather than waiting for the previous `avatar.media.ready` playback acknowledgement. Handle rejected media-element `play()` promises and provide a user action to start playback. An open data channel does not establish that the browser is playing the media; autoplay restrictions can prevent the user from hearing the beginning of the greeting.

## Configure ICE servers

Telnyx relays the call's media only through Telnyx TURN, and Telnyx TURN relays only between Telnyx addresses. Configure the client to relay through Telnyx TURN with `iceTransportPolicy: "relay"`. Direct connections and third-party TURN servers do not connect.

```js theme={null}
const pc = new RTCPeerConnection({
  iceServers: [
    {
      urls: [
        "turn:turn.telnyx.com:3478?transport=udp",
        "turn:turn.telnyx.com:3478?transport=tcp",
      ],
      username: TURN_USERNAME,
      credential: TURN_CREDENTIAL,
    },
  ],
  iceTransportPolicy: "relay",
});
```

Telnyx TURN requires credentials. See [STUN/TURN servers](https://developers.telnyx.com/docs/voice/sip-trunking/network-configuration/stun-turn-servers.md) for the endpoints and how to obtain credentials. These credentials authenticate the browser to the relay; they are separate from the Telnyx API key used by the application server.

The configuration above requires access to `turn.telnyx.com` on UDP or TCP port 3478. TCP on port 3478 is not an HTTPS/443 fallback. If a firewall or VPN blocks both transports, the call cannot connect even when HTTPS requests succeed. Handle ICE gathering and connection failures in the client, and release local media tracks when setup fails.

## Create the offer

Add the tracks and the data channel before creating the offer, then wait for ICE gathering to complete: Telnyx does not accept trickled candidates, so the offer must carry them.

```js theme={null}
const mic = await navigator.mediaDevices.getUserMedia({
  audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true },
});
pc.addTrack(mic.getAudioTracks()[0], mic);

// Receive the avatar's video. Omit for a voice-only call.
pc.addTransceiver("video", { direction: "recvonly" });

const events = pc.createDataChannel("oai-events");
events.onmessage = (message) => handleEvent(JSON.parse(message.data));

// One stream per element: the audio and video tracks can share a remote
// stream, and the video element must not play the audio a second time.
pc.ontrack = ({ track }) => {
  const element = track.kind === "video" ? avatarVideo : assistantAudio;
  element.srcObject = new MediaStream([track]);
};

const iceGatheringComplete = (pc) =>
  new Promise((resolve) => {
    if (pc.iceGatheringState === "complete") return resolve();
    pc.addEventListener("icegatheringstatechange", () => {
      if (pc.iceGatheringState === "complete") resolve();
    });
  });

try {
  await pc.setLocalDescription(await pc.createOffer());
  await iceGatheringComplete(pc);

  const response = await fetch("/session", {
    method: "POST",
    headers: { "Content-Type": "application/sdp" },
    body: pc.localDescription.sdp,
  });
  if (!response.ok) throw new Error(await response.text());
  await pc.setRemoteDescription({ type: "answer", sdp: await response.text() });
} catch (error) {
  // Release the microphone (and camera) when the call cannot be set up.
  pc.getSenders().forEach((sender) => sender.track?.stop());
  pc.close();
  throw error;
}
```

The offer must include:

| Element | Requirement |
| - | - |
| Audio | One audio m-line with Opus, carrying the microphone. The assistant's voice returns on the same m-line. |
| Data channel | A data channel labeled `oai-events`, created by the client. Other data channels are ignored. |
| Video | Optional, one video m-line with VP8. Use `recvonly` to receive an avatar, `sendrecv` to also send the camera, or `sendonly` to send the camera without an avatar. |

All m-lines must share one transport (`a=group:BUNDLE`), as browsers do by default.

## Create the call

The application server forwards the offer with its Telnyx API key:

```text theme={null}
POST https://api.telnyx.com/v2/ai/assistants/{assistant_id}/calls
Authorization: Bearer YOUR_TELNYX_API_KEY
```

Optional query parameter:

| Parameter | Description |
| - | - |
| `avatar` | Set to `false` to start the call without the assistant's [avatar](#show-an-avatar), even when the offer includes a video m-line that could receive it. Default: `true`. |

Send the offer as a `multipart/form-data` body with an `sdp` field, or as a raw `application/sdp` body. The multipart form also accepts an optional `session` field to pass [dynamic variables](https://developers.telnyx.com/docs/inference/ai-assistants/dynamic-variables.md) for this conversation:

```json theme={null}
{
  "assistant": {
    "dynamic_variables": { "customer_name": "Ada", "plan": "premium" }
  }
}
```

A successful request returns `201 Created` with the SDP answer as an `application/sdp` body. Return the answer to the browser unchanged.

Every request to the relay starts a call billed to the Telnyx account. Authenticate and authorize the browser's user before forwarding the offer, and rate-limit calls per user.

```js theme={null}
// Node.js (Express): relay the browser's offer to Telnyx.
// requireUser is the application's own authentication middleware.
app.post("/session", requireUser, express.text({ type: "application/sdp" }), async (req, res) => {
  const form = new FormData();
  form.set("sdp", req.body);
  form.set("session", JSON.stringify({ assistant: { dynamic_variables: { customer_name: "Ada" } } }));

  const telnyx = await fetch(
    `https://api.telnyx.com/v2/ai/assistants/${process.env.ASSISTANT_ID}/calls`,
    { method: "POST", headers: { Authorization: `Bearer ${process.env.TELNYX_API_KEY}` }, body: form },
  );

  res.status(telnyx.status).type(telnyx.headers.get("content-type") ?? "text/plain").send(await telnyx.text());
});
```

Client-supplied dynamic variables override same-named assistant defaults or values returned by the dynamic-variable webhook. Unspecified variables keep those values.

### Errors

Errors return a JSON error object with `error.code`:

| Code | HTTP status | Meaning |
| - | - | - |
| `invalid_offer` | 400 | The SDP offer is missing, larger than 65,536 bytes, or cannot be negotiated, for example without Opus audio. |
| `invalid_session` | 400 | `session` is not a JSON object. |
| `invalid_avatar` | 400 | The `avatar` query parameter is not `true` or `false`. |
| `unauthorized` | 401 | The API key is missing or invalid. |
| `assistant_not_found` | 404 | The assistant does not exist or is not available to this account. |
| `data_locality_mismatch` | 422 | This region cannot serve the assistant's in-transit data locality. |
| `in_transit_data_locality_unsupported` | 422 | In-transit data locality is not supported for this session. |
| `webrtc_unavailable` | 503 | Telnyx could not set up media for the call. Retry later. |

## Exchange events on the data channel

The `oai-events` data channel carries the same JSON events as the [conversation WebSocket](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md), with these differences:

| WebSocket | WebRTC |
| - | - |
| `input_audio_buffer.append` frames carry microphone audio. | The microphone track carries the audio. Do not send `input_audio_buffer.append`. |
| `response.output_audio.delta` frames carry the assistant's voice. | The audio track carries the voice. The data channel receives `response.output_audio.done` but no audio deltas. |
| The first `session.update` frame passes dynamic variables. | The `session` field of the call request passes them. Do not send `session.update`. |
| `session.created` reports the negotiated PCM formats. | `session.created` arrives when the conversation starts; audio formats are negotiated in SDP. |

Every other event works as on the WebSocket, including speech detection and transcripts, [text turns](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md#send-text-instead-of-audio), [interruption and barge-in](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md#interruption-and-barge-in), `response.cancel`, [tool calls](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md#tool-calls) and [vision frames](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md#add-vision-with-camera-or-screen-share-frames). Keep each serialized message below **1 MiB** and within the connection's negotiated [`pc.sctp.maxMessageSize`](https://www.w3.org/TR/webrtc/#dom-rtcsctptransport-maxmessagesize). The SCTP limit can be smaller than the Telnyx event limit. Measure the UTF-8 bytes of the entire JSON string, including any base64 image and metadata, before calling `send()`.

The assistant's voice must stream PCM at 8, 12, 16, 24 or 48 kHz. Otherwise the call starts with an `error` event on the data channel:

| Code | Meaning |
| - | - |
| `unsupported_voice_output_format` | The assistant's voice streams a compressed format, for example mp3 or opus. Configure a PCM voice. |
| `unsupported_voice_output_rate` | The assistant's voice streams PCM at a rate WebRTC cannot play, for example 22.05 or 44.1 kHz. Configure a voice at 8, 12, 16, 24 or 48 kHz. |

## Show an avatar

**Beta:** AI Assistant avatars are in beta. See the [Avatars guide](https://developers.telnyx.com/docs/inference/ai-assistants/avatars.md) for configuration, voice compatibility and pricing.

When the assistant has saved `avatar_settings` and the offer includes a video m-line that can receive VP8 (`recvonly` or `sendrecv`), the avatar's video arrives on that video track and its voice on the audio track, already synchronized. Play both. An offer without a receivable video m-line starts a voice-only call, even when the assistant has an avatar. To keep a receivable video m-line, for example to [send the camera](#send-the-camera), without the avatar, create the call with `?avatar=false`; a video m-line [added during the call](#add-video-during-a-call) then does not start the avatar either.

The data channel reports the avatar's state with `avatar.updated`:

```json theme={null}
{ "type": "avatar.updated", "state": "active" }
```

| `state` | Meaning |
| - | - |
| `starting` | The avatar is being prepared. |
| `active` | The avatar's video and voice are playing. |
| `unavailable` | The avatar could not start or stopped. The call continues with the assistant's voice on the same audio track; no action is needed besides hiding the video. |
| `off` | The call has no avatar. |

A runtime avatar failure also sends an `avatar.error` event with `error.fatal: false`. The conversation continues; the voice switches back on the same audio track automatically.

## Send the camera

To let the assistant see the caller, send the camera on the video m-line instead of `recvonly`. Add the camera track before creating the offer, in place of the `recvonly` transceiver:

```js theme={null}
const camera = await navigator.mediaDevices.getUserMedia({ video: true });
pc.addTrack(camera.getVideoTracks()[0], camera);
```

The m-line is then `sendrecv`: the camera goes up and the avatar, if any, comes down on the same m-line. Telnyx samples the camera every 2 seconds into a JPEG of at most 1280 pixels wide, tagged `camera`, and buffers it for the [vision tool](https://developers.telnyx.com/docs/inference/ai-assistants/vision.md). The assistant needs the `vision` tool and a vision-capable model; buffered frames do not trigger responses on their own. Image frames sent on the data channel are buffered the same way.

For camera or screen snapshots sent as image events, reduce JPEG quality or image dimensions until the complete message fits both data-channel limits. Reducing quality or dimensions can make small text and fine detail harder to interpret. These message-size limits apply to image events, not to the native camera RTP track described above.

Send only while the data channel is open, monitor `bufferedAmount` to avoid accumulating stale snapshots, and handle `send()` exceptions. If a frame cannot fit or be sent, surface the failure in the application's camera-sharing UI; reducing the image and retrying are client responsibilities. A successful `send()` queues the message locally and does not confirm that the assistant has used the image.

## Add video during a call

A call that started without a video m-line can add one. Add the transceiver, set the new offer as the local description, wait for ICE gathering, and send it on the data channel as a `webrtc.offer` event. Telnyx replies with a `webrtc.answer` event; apply it with `setRemoteDescription`:

```js theme={null}
pc.addTransceiver("video", { direction: "recvonly" });
await pc.setLocalDescription(await pc.createOffer());
await iceGatheringComplete(pc);
events.send(JSON.stringify({ type: "webrtc.offer", sdp: pc.localDescription.sdp }));

// In the data channel's message handler:
if (event.type === "webrtc.answer") {
  await pc.setRemoteDescription({ type: "answer", sdp: event.sdp });
}
```

If the assistant has an avatar, it starts once the new video m-line can receive it. Send one offer at a time and keep a single video m-line. A refused offer returns an `error` event instead:

| Code | Meaning |
| - | - |
| `invalid_offer` | The offer was not applied. Roll back the local description; the call continues unchanged. |
| `renegotiation_in_progress` | A previous offer is still being answered. Wait for its answer. |
| `renegotiation_failed` | The offer was applied but could not be answered. The two sides no longer agree; end the call. |

## End the call

Close the peer connection to end the call. Telnyx ends the conversation as soon as the `oai-events` data channel closes. When a client disappears without closing, for example after a network loss, Telnyx ends the conversation within seconds once media stops arriving.

```js theme={null}
// Stop every local track (microphone and, if sent, camera), then close.
pc.getSenders().forEach((sender) => sender.track?.stop());
pc.close();
```

The data channel must open within 20 seconds of the call request; otherwise Telnyx ends the call.