> ## Documentation Index
> Fetch the complete documentation index at: https://developers.telnyx.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime voice conversations over WebSocket

> Open a Telnyx AI Assistant WebSocket: stream microphone audio, receive transcripts and synthesized speech, share camera or screen snapshots with vision, handle barge-in, and answer tool calls.

Open a real-time voice conversation with a [Telnyx AI Assistant](https://portal.telnyx.com/#/ai/assistants) over a single WebSocket. Your application streams microphone audio to Telnyx as base64-encoded PCM16 frames, and Telnyx streams back lifecycle events, speech-detection events, user and assistant transcripts, and the assistant's synthesized speech.

Everything about the assistant's behavior — instructions, model, voice, language, tools, transcription and interruption settings — is configured on the assistant itself, in the [Portal](https://portal.telnyx.com/#/ai/assistants) or through the Assistants API. The WebSocket carries audio, [image frames for vision](#add-vision-with-camera-or-screen-share-frames), conversation events, and per-conversation [dynamic variables](#pass-dynamic-variables). There is no frame to request a response — the assistant owns turn-taking and answers automatically when the user finishes speaking.

> [!NOTE]
>
> For browser and mobile clients, and for [avatar](https://developers.telnyx.com/docs/inference/ai-assistants/avatars.md) conversations, use [WebRTC](https://developers.telnyx.com/docs/inference/ai-assistants/webrtc-conversations.md). It carries the same conversation events, with audio and video as media tracks.

## How a conversation flows

```
Client → Telnyx   (connect wss://.../v2/ai/assistants/{assistant_id}/conversation)
Client → Telnyx   {"type":"session.update","session":{...}}          (optional)
Client ← Telnyx   {"type":"session.created", ...}
Client → Telnyx   {"type":"input_audio_buffer.append","audio":"<base64 pcm16>"}
Client → Telnyx   {"type":"input_audio_buffer.append","audio":"<base64 pcm16>"}
Client ← Telnyx   {"type":"input_audio_buffer.speech_started"}
Client ← Telnyx   {"type":"input_audio_buffer.speech_stopped"}
Client ← Telnyx   {"type":"conversation.item.input_audio_transcription.completed","transcript":"..."}
Client ← Telnyx   {"type":"response.created","response":{"id":"resp_9c1f4a2b"}}
Client ← Telnyx   {"type":"response.output_audio.delta","delta":"<base64 pcm16>", ...}
Client ← Telnyx   {"type":"response.output_audio_transcript.delta","delta":"We are open ", ...}
Client ← Telnyx   {"type":"response.output_audio.done","response_id":"resp_9c1f4a2b"}
Client ← Telnyx   {"type":"response.done","response":{"id":"resp_9c1f4a2b","status":"completed"}}
```

1. Open a WebSocket connection, optionally setting the audio format through query parameters.
2. Optionally send a `session.update` frame as your first message to pass [dynamic variables](#pass-dynamic-variables) for the conversation.
3. Telnyx authenticates the connection, starts a conversation with the requested assistant, and sends a `session.created` frame confirming the negotiated input and output audio formats.
4. Stream microphone audio as `input_audio_buffer.append` frames.
5. Telnyx runs server-side voice-activity detection (VAD) and sends `input_audio_buffer.speech_started` and `input_audio_buffer.speech_stopped` on speech edges, followed by the user's transcript.
6. The assistant responds with a `response.created` frame, a stream of audio and transcript deltas, then `response.output_audio.done` and `response.done`.
7. Speaking while the assistant is talking triggers barge-in: Telnyx stops the response and starts a new turn.

## Connect and authenticate

```
wss://api.telnyx.com/v2/ai/assistants/{assistant_id}/conversation
```

Authenticate the WebSocket handshake with a Telnyx API v2 key in the `Authorization` header:

```
Authorization: Bearer YOUR_TELNYX_API_KEY
```

> [!NOTE]
>
> Connect from a server-side WebSocket client that supports the `Authorization` header. Keep the API key on your server. The examples in this guide use Node.js and Python.

Set audio preferences at connect time through query parameters:

| Parameter | Description |
| - | - |
| `input_sample_rate` | Sample rate, in Hz, of the PCM16 audio you stream to Telnyx. One of `8000`, `16000`, `24000`, `44100`, `48000`. Default: `16000`. |
| `input_format` | Encoding of the audio you stream to Telnyx. Only `pcm16` is supported (the default). |
| `output_format` | Encoding of the assistant audio Telnyx streams back. Only `pcm16` is supported (the default). |
| `output_sample_rate` | Preferred sample rate, in Hz, for the assistant audio. Advisory only — the effective output rate is determined by the assistant's voice and reported in `session.created`. |

```javascript Node.js theme={null}
import WebSocket from "ws";

const ASSISTANT_ID = "assistant-0f4e8b2a";
const url = `wss://api.telnyx.com/v2/ai/assistants/${ASSISTANT_ID}/conversation?input_sample_rate=16000`;

const ws = new WebSocket(url, {
  headers: { Authorization: `Bearer ${process.env.TELNYX_API_KEY}` },
});

ws.on("open", () => console.log("Connected"));
```

```python Python theme={null}
import asyncio
import os
import websockets

ASSISTANT_ID = "assistant-0f4e8b2a"
URL = f"wss://api.telnyx.com/v2/ai/assistants/{ASSISTANT_ID}/conversation?input_sample_rate=16000"

async def main():
    headers = {"Authorization": f"Bearer {os.environ['TELNYX_API_KEY']}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        print("Connected")

asyncio.run(main())
```

All frames in both directions are JSON text messages with a `type` field. Audio travels inside JSON frames as base64 — never as binary WebSocket frames.

## Session lifecycle

The first session frame is `session.created`. It confirms the conversation has started and reports the negotiated audio formats — read the output rate from it before playing PCM audio:

```json theme={null}
{
  "type": "session.created",
  "session": {
    "conversation_id": "3E6F995F-85F7-4705-9741-53B116D28237",
    "assistant_id": "assistant-0f4e8b2a",
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 16000 },
        "turn_detection": { "type": "server_vad" }
      },
      "output": {
        "format": { "type": "audio/pcm", "rate": 24000 }
      }
    }
  }
}
```

Each WebSocket connection starts a new conversation with the assistant. Turn detection is always `server_vad` — there is no manual turn mode to configure.

> [!WARNING]
>
> Only assistants whose voice streams raw PCM are supported over this WebSocket. Assistants whose voice outputs a compressed format (for example mp3 or opus) are rejected with an `unsupported_voice_output_format` error.

## Pass dynamic variables

If the assistant's instructions, greeting, or tools use [dynamic variables](https://developers.telnyx.com/docs/inference/ai-assistants/dynamic-variables.md), supply the values for this conversation with a `session.update` frame. Send it as your **first** frame, immediately on `open` — before any audio, and without waiting for `session.created`:

```json theme={null}
{
  "type": "session.update",
  "session": {
    "assistant": {
      "dynamic_variables": {
        "customer_name": "Ada",
        "account_tier": "pro"
      }
    }
  }
}
```

Telnyx starts the conversation as soon as the frame arrives; variables are resolved before the assistant's first word. Omitting `session.update` starts the conversation with no client-supplied variables after a short timeout.

Client-supplied values override same-named assistant defaults or values returned by the configured dynamic-variable webhook. Unspecified variables retain those defaults or webhook values; an empty object does not clear them.

Only `type` is required. A bare `{ "type": "session.update" }` — or one whose `session` object carries no `dynamic_variables` — is valid: it skips that timeout and starts the conversation with no client-supplied variables.

```javascript Node.js theme={null}
ws.on("open", () => {
  ws.send(
    JSON.stringify({
      type: "session.update",
      session: {
        assistant: {
          dynamic_variables: { customer_name: "Ada", account_tier: "pro" },
        },
      },
    })
  );
});
```

```python Python theme={null}
async with websockets.connect(URL, additional_headers=headers) as ws:
    await ws.send(json.dumps({
        "type": "session.update",
        "session": {
            "assistant": {
                "dynamic_variables": {"customer_name": "Ada", "account_tier": "pro"},
            },
        },
    }))
```

Values are string key/value pairs, and they reach the assistant exactly like dynamic variables passed on the REST conversation API.

> [!NOTE]
>
> `session.update` configures nothing but dynamic variables, and only before the conversation starts. Sending it after `session.created` is rejected with a `session_update_after_start` error, and a `dynamic_variables` value that is not an object is rejected with `invalid_session_update`. The `telnyx_` namespace is reserved for [system variables](https://developers.telnyx.com/docs/inference/ai-assistants/dynamic-variables.md#telnyx-system-variables), so `telnyx_conversation_channel` is set by Telnyx and cannot be overridden.

## Handling audio

### Stream microphone audio

Send audio as `input_audio_buffer.append` frames. The `audio` field is base64-encoded, raw little-endian PCM16 (16-bit signed, mono) at the sample rate you chose with `input_sample_rate`:

```json theme={null}
{ "type": "input_audio_buffer.append", "audio": "<base64-encoded PCM16>" }
```

There is no commit step. Keep appending audio continuously — server-side VAD detects when the user starts and stops speaking and turns the buffered audio into conversation turns automatically.

A few practical rules:

* **Stream at a real-time pace**, in small chunks (for example 20–100 ms of audio per frame). The server enforces an ingress budget; sending far faster than real time fails with an `ingress_budget_exceeded` error.
* **Keep frames under 1 MiB.** Larger frames are rejected with a `frame_too_large` error.
* **Wait for `session.created`** before sending audio. `session.update` is the only frame sent before it.

```javascript Node.js theme={null}
import fs from "node:fs";

const SAMPLE_RATE = 16000;
const CHUNK_MS = 100;
const CHUNK_BYTES = (SAMPLE_RATE * 2 * CHUNK_MS) / 1000; // 16-bit mono

// speech.raw is raw little-endian PCM16 at 16 kHz
const pcm = fs.readFileSync("speech.raw");
let offset = 0;

const timer = setInterval(() => {
  if (offset >= pcm.length) {
    clearInterval(timer);
    return;
  }
  const chunk = pcm.subarray(offset, offset + CHUNK_BYTES);
  offset += CHUNK_BYTES;
  ws.send(
    JSON.stringify({
      type: "input_audio_buffer.append",
      audio: chunk.toString("base64"),
    })
  );
}, CHUNK_MS);
```

```python Python theme={null}
import base64
import json

SAMPLE_RATE = 16000
CHUNK_MS = 100
CHUNK_BYTES = SAMPLE_RATE * 2 * CHUNK_MS // 1000  # 16-bit mono

async def stream_audio(ws):
    # speech.raw is raw little-endian PCM16 at 16 kHz
    with open("speech.raw", "rb") as f:
        while chunk := f.read(CHUNK_BYTES):
            await ws.send(json.dumps({
                "type": "input_audio_buffer.append",
                "audio": base64.b64encode(chunk).decode(),
            }))
            await asyncio.sleep(CHUNK_MS / 1000)  # real-time pacing
```

If your capture pipeline produces float samples (for example the Web Audio API), convert them to PCM16 before encoding:

```javascript theme={null}
function floatTo16BitPCM(float32Array) {
  const buffer = new ArrayBuffer(float32Array.length * 2);
  const view = new DataView(buffer);
  for (let i = 0; i < float32Array.length; i++) {
    const s = Math.max(-1, Math.min(1, float32Array[i]));
    view.setInt16(i * 2, s < 0 ? s * 0x8000 : s * 0x7fff, true);
  }
  return Buffer.from(buffer);
}
```

### Speech detection and user transcripts

Telnyx runs voice-activity detection server-side and reports speech edges. Both events are edge-triggered — each is sent only when the speaking state actually changes, and `speech_stopped` is never sent without a preceding `speech_started`:

```json theme={null}
{ "type": "input_audio_buffer.speech_started" }
{ "type": "input_audio_buffer.speech_stopped" }
```

After the user's turn ends, Telnyx sends the transcript of what they said:

```json theme={null}
{
  "type": "conversation.item.input_audio_transcription.completed",
  "transcript": "What are your business hours?"
}
```

### Receive assistant audio

Each assistant turn arrives as an ordered sequence of frames, correlated by `response_id`:

| Frame | Meaning |
| - | - |
| `response.created` | The assistant turn started. `response.id` correlates everything that follows. |
| `response.output_audio.delta` | A chunk of synthesized speech — base64 PCM16 at the output rate from `session.created`. |
| `response.output_audio_transcript.delta` | A chunk of the text transcript of the spoken response, streamed alongside the audio. |
| `response.output_audio.done` | The assistant finished streaming audio for the turn. |
| `response.done` | The turn is over, with a final `status` of `completed` or `cancelled`. |

```javascript Node.js theme={null}
let outputRate = 24000;
const playback = [];

ws.on("message", async (raw) => {
  const event = JSON.parse(raw.toString());

  switch (event.type) {
    case "session.created":
      outputRate = event.session.audio.output.format.rate;
      break;

    case "conversation.item.input_audio_transcription.completed":
      console.log(`You said: ${event.transcript}`);
      break;

    case "response.output_audio.delta":
      // Feed to your audio output as it arrives for lowest latency
      playback.push(Buffer.from(event.delta, "base64"));
      break;

    case "response.output_audio_transcript.delta":
      process.stdout.write(event.delta);
      break;

    case "response.done":
      console.log(`\n[turn ${event.response.id}: ${event.response.status}]`);
      break;

    case "error":
      console.error(`${event.error.code}: ${event.error.message}`);
      break;
  }
});
```

```python Python theme={null}
import base64
import json

async def handle_events(ws):
    output_rate = 24000
    playback = bytearray()

    async for raw in ws:
        event = json.loads(raw)
        match event["type"]:
            case "session.created":
                output_rate = event["session"]["audio"]["output"]["format"]["rate"]

            case "conversation.item.input_audio_transcription.completed":
                print(f"You said: {event['transcript']}")

            case "response.output_audio.delta":
                # Feed to your audio output as it arrives for lowest latency
                playback.extend(base64.b64decode(event["delta"]))

            case "response.output_audio_transcript.delta":
                print(event["delta"], end="", flush=True)

            case "response.done":
                print(f"\n[turn {event['response']['id']}: {event['response']['status']}]")

            case "error":
                print(f"{event['error']['code']}: {event['error']['message']}")
```

For a natural conversation, play deltas as they arrive rather than waiting for `response.done`. Buffer the decoded PCM16 in a queue that your audio output drains at the output sample rate.

## Interruption and barge-in

Barge-in is built in. If the user speaks while the assistant is talking, Telnyx detects it, stops generating the response, and starts a new turn — you don't send anything to make that happen. The interrupted turn finishes with `response.done` and `status: "cancelled"`.

One thing remains your responsibility: audio you have already received and queued locally. When you see `input_audio_buffer.speech_started` during playback, flush your local playback queue so the assistant doesn't keep talking out of your speakers over the user:

```javascript theme={null}
case "input_audio_buffer.speech_started":
  playback.length = 0; // drop queued assistant audio immediately
  break;
```

You can also interrupt programmatically — for example when the user taps a stop button — with `response.cancel`:

```json theme={null}
{ "type": "response.cancel" }
```

Include `response_id` to target a specific response, or omit it to cancel the current one. Cancelling flushes playback on the Telnyx side and is followed by a `response.done` frame with status `cancelled`. Cancelling a stale or already-finished response is a no-op.

Interruption sensitivity is tuned on the assistant, not the socket — see [Interruption Settings](https://developers.telnyx.com/docs/inference/ai-assistants/interruption-settings.md).

## Send text instead of audio

Inject a completed user turn as text with `conversation.item.create`. The assistant answers it exactly as it would a spoken turn — including responding with audio:

```json theme={null}
{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [{ "type": "input_text", "text": "What are your business hours?" }]
  }
}
```

There is no `response.create` frame — the assistant owns turn-taking and answers automatically.

Client-side tool results use a `function_call_output` item (see [Client-side tools](#client-side-tools-execute-and-respond)). Unsupported item shapes are rejected with an `invalid_item` error.

## Add vision with camera or screen-share frames

**Beta:** The [vision tool](https://developers.telnyx.com/docs/inference/ai-assistants/vision.md) lets an assistant inspect camera or screen-share snapshots during the same realtime conversation. For example, a user can show a product or share an error message while asking the assistant for help. Your client captures and sends individual images; a continuous video stream is not sent over this WebSocket.

Vision does not require an avatar.

Add `{ "type": "vision" }` to the assistant's `tools`. Either the base model or its enabled delegation model must support image input. S2S assistants, including GPT Live, can inspect buffered frames through a vision-capable delegation model. Configure the tool and model before connecting; see [Vision model support](https://developers.telnyx.com/docs/inference/ai-assistants/vision.md#model-support).

You can also start with the **Avatar with vision** template in the [Portal](https://portal.telnyx.com/#/ai/assistants) for a configured assistant. Your WebSocket client still needs to capture and send image frames.

After `session.created`, send snapshots as `conversation.item.create` frames with `"buffer": true` and only `input_image` content:

```json theme={null}
{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "buffer": true,
    "content": [{ "type": "input_image", "image_url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ..." }]
  }
}
```

Use a JPEG, PNG, or WebP base64 data URL, or an HTTPS image URL. The base64 content above is abbreviated. Optional `item.captured_at` records the capture time in ISO 8601 format; if omitted, Telnyx uses the receipt time. Image items require `buffer: true`; send text prompts as separate items.

Optional `item.tags` labels the image source, for example `"tags": ["camera"]` or `"tags": ["screenshare"]`. Tags apply to every image in the item. Send up to 5 tags, each 1–64 characters; `untagged` is reserved for frames sent without tags. Frames are buffered per tag. See [Tag frames](https://developers.telnyx.com/docs/inference/ai-assistants/vision.md#tag-frames) for tag matching and multiple-source examples.

The `vision` function accepts `action: "look"` (the default) or `action: "list"`, and an optional `tag`. A look without `tag` retrieves the most recent frame of any tag; a look with `tag` retrieves the most recent matching frame. Listing returns available tags and when each last received a frame, without showing an image. For an untagged demo, instruct the assistant to call `vision` with `action: "look"` and omit `tag`; it does not need to list tags first.

Continue streaming microphone audio or [send a text turn](#send-text-instead-of-audio), such as "What does this error message mean?" Buffered images do not trigger responses. The assistant decides when to call `vision` and inspect recent frames while answering. Frames older than `max_frame_age_secs` (10 seconds by default, measured from receipt) are ignored. Keep sending frames for static content too; freshness applies per frame, including when selecting a tag. Buffered images are held in memory for the conversation and are not added to conversation history, transcripts, or webhooks.

Downscale snapshots before sending, for example to 640×480 JPEG, and start with one frame every 1–5 seconds. Keep the entire JSON message below **1 MiB**, including base64 overhead. Oversized messages return `frame_too_large`; malformed buffered items return `invalid_frame` without closing the socket.

See the [Vision guide](https://developers.telnyx.com/docs/inference/ai-assistants/vision.md) for tool settings, client capture examples, other transports, and retention details, or the [conversation WebSocket reference](https://developers.telnyx.com/api-reference/websockets/talk-to-an-ai-assistant-over-websocket.md) for the complete frame schema.

## Tool calls

Assistants can use two kinds of tools during a realtime conversation. Both are configured on the assistant — see the [Tools Library](https://developers.telnyx.com/docs/inference/ai-assistants/tools-library.md).

### Server-side tools: observe

Webhook and MCP tools execute on Telnyx. The socket surfaces them as informational frames so you can show tool activity in your UI, but you don't execute or respond to them:

```json theme={null}
{
  "type": "response.tool_call.started",
  "tool_call": { "id": "call_7a3f21b8", "name": "get_business_hours", "arguments": "{\"location\":\"downtown\"}" }
}
```

```json theme={null}
{
  "type": "response.tool_call.completed",
  "tool_call": { "id": "call_7a3f21b8", "name": "get_business_hours" },
  "status": "success"
}
```

### Client-side tools: execute and respond

When the assistant invokes a [client-side tool](https://developers.telnyx.com/docs/inference/ai-assistants/client-side-tools.md), Telnyx sends a `conversation.item.created` frame containing a `function_call` item. Run the tool locally, then return the result with a `conversation.item.create` frame carrying a `function_call_output` item that references the same `call_id`. The assistant continues once the result arrives, or after the tool times out.

```javascript Node.js theme={null}
case "conversation.item.created": {
  const item = event.item;
  if (item.type === "function_call") {
    const args = JSON.parse(item.arguments);
    const result = await runTool(item.name, args); // your implementation
    ws.send(
      JSON.stringify({
        type: "conversation.item.create",
        item: {
          type: "function_call_output",
          call_id: item.call_id,
          output: JSON.stringify(result),
        },
      })
    );
  }
  break;
}
```

```python Python theme={null}
case "conversation.item.created":
    item = event["item"]
    if item["type"] == "function_call":
        args = json.loads(item["arguments"])
        result = await run_tool(item["name"], args)  # your implementation
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {
                "type": "function_call_output",
                "call_id": item["call_id"],
                "output": json.dumps(result),
            },
        }))
```

## Error handling

Errors arrive as `error` frames. Some errors are non-fatal and leave the session open; others close the connection after the frame is sent.

```json theme={null}
{
  "type": "error",
  "error": { "code": "invalid_audio", "message": "Invalid base64-encoded audio payload" }
}
```

| Code | Meaning |
| - | - |
| `session_not_ready` | A frame arrived before the session was ready. Wait for `session.created`. |
| `frame_too_large` | A frame exceeded the 1 MiB limit. Send smaller audio chunks. |
| `unsupported_event` | The `type` field is not a supported client frame. |
| `invalid_json` | The frame was not valid JSON. |
| `invalid_audio` | The `audio` field was not valid base64. |
| `invalid_item` | A `conversation.item.create` item had an unsupported shape. |
| `invalid_frame` | A buffered (`"buffer": true`) `conversation.item.create` item had a non-image part, an unsupported image URL, an invalid `captured_at`, or invalid tags. See [Vision](https://developers.telnyx.com/docs/inference/ai-assistants/vision.md). |
| `invalid_session_update` | A `session.update` frame carried a `dynamic_variables` value that was not an object. |
| `session_update_after_start` | A `session.update` frame arrived after the conversation had already started. Send it as your first frame. |
| `ingress_budget_exceeded` | Audio arrived faster than the session's ingress budget allows. Stream at a real-time pace. |
| `unsupported_voice_output_format` | The assistant's voice streams a compressed format (for example mp3 or opus), which this WebSocket does not support. |
| `conversation_start_failed` | Telnyx could not start a conversation with the requested assistant. |
| `conversation_ended` | The conversation has ended. |
| `session_idle_timeout` | The session was closed after a period of inactivity. |
| `session_max_duration_exceeded` | The session reached its maximum duration. |

If the connection closes, reconnecting starts a new conversation — a fresh `conversation_id` is issued in `session.created`.

## Learn more

* **Conversation WebSocket reference** — The full frame-by-frame reference, under **Assistants API → Conversation WebSocket** in the sidebar
* **[Vision](https://developers.telnyx.com/docs/inference/ai-assistants/vision.md)** — Configure the vision tool and stream camera or screen-share snapshots
* **[Client-Side Tools](https://developers.telnyx.com/docs/inference/ai-assistants/client-side-tools.md)** — Configure tools that execute in your application
* **[Transcription Settings](https://developers.telnyx.com/docs/inference/ai-assistants/transcription-settings.md)** and **[Interruption Settings](https://developers.telnyx.com/docs/inference/ai-assistants/interruption-settings.md)** — Tune how the assistant hears and yields
* **[Voice Assistant Quickstart](https://developers.telnyx.com/docs/inference/ai-assistants/no-code-voice-assistant.md)** — Create and configure an assistant in the Portal
