> ## Documentation Index
> Fetch the complete documentation index at: https://developers.telnyx.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Vision

> Analyze images sent through MMS or the chat widget, or use the BETA vision tool to inspect streamed camera and screen-share frames during live conversations.

AI Assistants use images as visual context through MMS, chat-widget messages, and live camera or screen-share snapshots. Each path uses a model that supports image input.

| Vision option | When to use it | Setup |
| - | - | - |
| MMS images | A caller sends a photo, screenshot, or document for the assistant to discuss during a voice call. | Enable messaging for the assistant and configure an image-capable model. See [MMS integration](/docs/inference/ai-assistants/no-code-voice-assistant#mms-integration-during-voice-calls). |
| Chat-widget images | A website visitor shares an image as part of a chat conversation. | Configure an image-capable model and use the image-message support in the [embedded widget](/docs/inference/ai-assistants/no-code-voice-assistant#embeddable-widget). |
| Vision tool (BETA) | A meeting bot or customer-service application provides a changing camera or screen-share view during a live conversation. | Add the `vision` tool to the assistant and connect a client or integration that captures and streams image snapshots. |

## Vision tool for live conversations

Use the `vision` tool when visual context changes as the conversation progresses: a customer demonstrates an equipment problem, a participant presents a document in a meeting, or a user shares a screen while troubleshooting. The client sends periodic snapshots, and the assistant chooses when to inspect a recent frame based on the conversation — for example, when the user asks "what am I holding?" or "can you read this error?"

Frames are buffered as they arrive, so the client can keep the view current without triggering an answer for every snapshot. The assistant retrieves visual context on demand by calling `vision`.

<Warning>
  **BETA:** The `vision` tool for buffered frames is in beta. Its configuration and behavior may change before general availability.
</Warning>

Setting up this path involves both the assistant and the media source: configure the tool, then have your client or integration capture and send snapshots through the [realtime conversation WebSocket](/docs/inference/ai-assistants/realtime-conversations), the Voice SDK, or the [assistant event stream](/docs/inference/ai-assistants/conversation-event-stream). The model receives individual image snapshots from the camera or screen share. The following sections cover the tool configuration, frame format, and transport examples.

## How buffered frames work

1. **Your client sends frames** with a `conversation.item.create` frame whose item has `"buffer": true` and `input_image` content. Telnyx buffers recent frames for the conversation, in memory only.
2. **Nothing else happens.** Buffered frames never start an assistant response and are never added to the conversation history, transcripts, or webhooks.
3. **The assistant decides when to look.** With the vision tool enabled, the assistant chooses when to look and which buffered frames it needs. Frames reach the model only when it looks.
4. **Stale frames are ignored.** Frames that Telnyx received more than `max_frame_age_secs` ago (10 seconds by default) are not shown to the assistant, so it never describes a frozen or disconnected camera as live.

<Note>
  The vision tool requires a model that supports image input. Saving an assistant with a vision tool on a Telnyx-hosted model without image support is rejected. The tool is not available to GPT-Live assistants or to [delegated](/docs/inference/ai-assistants/delegation) tool runs.
</Note>

## Enable the vision tool

Add a `vision` tool to the assistant. Every setting is optional:

```json theme={null}
{
  "tools": [
    { "type": "vision", "vision": { "max_frame_age_secs": 10 } }
  ]
}
```

| Field | Default | Description |
| - | - | - |
| `max_frame_age_secs` | `10` | Frames received longer ago than this are not shown to the assistant. Minimum `1`. |

`{ "type": "vision" }` on its own is valid. An assistant can have at most one vision tool. You can also add it in the [Portal](https://portal.telnyx.com/#/ai/assistants) under **Tools → Add Tool → Vision (BETA)**.

## Send frames

A buffered item is a user `message` with `"buffer": true` and one or more `input_image` parts:

```json theme={null}
{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "buffer": true,
    "captured_at": "2026-10-12T14:03:12.120Z",
    "content": [
      { "type": "input_image", "image_url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ..." }
    ]
  }
}
```

| Field | Required | Description |
| - | :-: | - |
| `buffer` | ✅ | Must be the boolean `true` for this buffered-frame protocol. Image items without it are not supported on these conversation sockets. |
| `content` | ✅ | One or more `input_image` parts and nothing else. `image_url` is a `data:image/jpeg`, `data:image/png` or `data:image/webp` base64 URL, or an `https` URL. |
| `captured_at` | | When the client captured the frame (ISO 8601). Defaults to when Telnyx received it. Shown to the assistant; freshness is measured from when Telnyx received the frame, so client clock skew does not matter. |

If any part is invalid — for example a text part, an unsupported image type, an `http` URL, or a malformed `captured_at` — the whole item is rejected and nothing is buffered.

The 1 MiB WebSocket limit applies to the entire serialized message, including base64 image data and JSON overhead. A message that exceeds this limit is rejected before buffered-item validation. The realtime socket reports `frame_too_large`; the assistant event-stream socket returns an error and closes the connection.

<Tip>
  Downscale frames before sending (for example 640×480 JPEG). Keep the complete message below 1 MiB; base64 encoding increases the image size by roughly one third before JSON overhead. Smaller images are also faster for the model to read. One frame every 1–5 seconds is plenty for most live conversations.
</Tip>

### WebRTC (Voice SDK)

Send frames over the active call with `sendAIConversationMessage`. The SDK forwards the item as-is; in TypeScript its published `AIConversationOutboundItem` type does not list buffered image items yet, so cast the item until it does:

```typescript theme={null}
const canvas = document.createElement('canvas');
canvas.width = 640;
canvas.height = 480;
const context = canvas.getContext('2d');
if (!context) throw new Error('2D canvas is unavailable');

setInterval(() => {
  context.drawImage(videoElement, 0, 0, canvas.width, canvas.height);

  call.sendAIConversationMessage({
    type: 'message',
    role: 'user',
    buffer: true,
    captured_at: new Date().toISOString(),
    content: [{ type: 'input_image', image_url: canvas.toDataURL('image/jpeg', 0.7) }],
  } as any); // until the SDK type includes buffered image items
}, 2000);
```

### Realtime WebSocket

On a [realtime conversation](/docs/inference/ai-assistants/realtime-conversations) socket, send the `conversation.item.create` frame shown above after `session.created`. For messages within the size limit, an invalid buffered item is answered with an `invalid_frame` error; the socket stays open.

### Assistant event stream

On the customer-configured [conversation event stream](/docs/inference/ai-assistants/conversation-event-stream) socket, send the same `conversation.item.create` frame. An invalid buffered item is answered with an `error` frame. Messages exceeding the 1 MiB transport limit also close the connection.

## Privacy

The following describes the buffered-frame path. It does not describe storage or retention of MMS attachments or chat-widget image messages.

The vision tool buffers image frames in memory for the running conversation and discards the buffer when it ends. Buffered images are not added to conversation history, transcripts, or webhooks, and reach the model only when the assistant invokes the tool.

This does not disable or change recording. Separate video recordings — including recordings made through [Telnyx Meeting](/docs/meeting/recording) — can retain the visual content shown during the conversation. Those recordings are stored and managed separately from the vision buffer. Telnyx Meeting sessions record automatically; see [Collect Results](/docs/meeting/collect-results#recordings) for recording retrieval and deletion.