Vision

AI Assistants use images as visual context through MMS, chat-widget messages, and live camera or screen-share snapshots. Each path uses a model that supports image input.

Vision option	When to use it	Setup
MMS images	A caller sends a photo, screenshot, or document for the assistant to discuss during a voice call.	Enable messaging for the assistant and configure an image-capable model. See MMS integration.
Chat-widget images	A website visitor shares an image as part of a chat conversation.	Configure an image-capable model and use the image-message support in the embedded widget.
Vision tool (BETA)	A meeting bot or customer-service application provides a changing camera or screen-share view during a live conversation.	Add the `vision` tool to the assistant and connect a client or integration that captures and streams image snapshots.

Vision tool for live conversations

Use the vision tool when visual context changes as the conversation progresses: a customer demonstrates an equipment problem, a participant presents a document in a meeting, or a user shares a screen while troubleshooting. The client sends periodic snapshots, and the assistant chooses when to inspect a recent frame based on the conversation — for example, when the user asks “what am I holding?” or “can you read this error?” Frames are buffered as they arrive, so the client can keep the view current without triggering an answer for every snapshot. The assistant retrieves visual context on demand by calling vision.

BETA: The vision tool for buffered frames is in beta. Its configuration and behavior may change before general availability.

Setting up this path involves both the assistant and the media source: configure the tool, then have your client or integration capture and send snapshots through the realtime conversation WebSocket, the Voice SDK, or the assistant event stream. The model receives individual image snapshots from the camera or screen share. The following sections cover the tool configuration, frame format, and transport examples.

How buffered frames work

Your client sends frames with a conversation.item.create frame whose item has "buffer": true and input_image content. Telnyx buffers recent frames for the conversation, in memory only.
Nothing else happens. Buffered frames never start an assistant response and are never added to the conversation history, transcripts, or webhooks.
The assistant decides when to look. With the vision tool enabled, the assistant chooses when to look and which buffered frames it needs. Frames reach the model only when it looks.
Stale frames are ignored. Frames that Telnyx received more than max_frame_age_secs ago (10 seconds by default) are not shown to the assistant, so it never describes a frozen or disconnected camera as live.

The vision tool requires a model that supports image input. Saving an assistant with a vision tool on a Telnyx-hosted model without image support is rejected. The tool is not available to GPT-Live assistants or to delegated tool runs.

Enable the vision tool

Add a vision tool to the assistant. Every setting is optional:

{
  "tools": [
    { "type": "vision", "vision": { "max_frame_age_secs": 10 } }
  ]
}

Field	Default	Description
`max_frame_age_secs`	`10`	Frames received longer ago than this are not shown to the assistant. Minimum `1`.

{ "type": "vision" } on its own is valid. An assistant can have at most one vision tool. You can also add it in the Portal under Tools → Add Tool → Vision (BETA).

Send frames

A buffered item is a user message with "buffer": true and one or more input_image parts:

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "buffer": true,
    "captured_at": "2026-10-12T14:03:12.120Z",
    "content": [
      { "type": "input_image", "image_url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ..." }
    ]
  }
}

Field	Required	Description
`buffer`	✅	Must be the boolean `true` for this buffered-frame protocol. Image items without it are not supported on these conversation sockets.
`content`	✅	One or more `input_image` parts and nothing else. `image_url` is a `data:image/jpeg`, `data:image/png` or `data:image/webp` base64 URL, or an `https` URL.
`captured_at`		When the client captured the frame (ISO 8601). Defaults to when Telnyx received it. Shown to the assistant; freshness is measured from when Telnyx received the frame, so client clock skew does not matter.

If any part is invalid — for example a text part, an unsupported image type, an http URL, or a malformed captured_at — the whole item is rejected and nothing is buffered. The 1 MiB WebSocket limit applies to the entire serialized message, including base64 image data and JSON overhead. A message that exceeds this limit is rejected before buffered-item validation. The realtime socket reports frame_too_large; the assistant event-stream socket returns an error and closes the connection.

Downscale frames before sending (for example 640×480 JPEG). Keep the complete message below 1 MiB; base64 encoding increases the image size by roughly one third before JSON overhead. Smaller images are also faster for the model to read. One frame every 1–5 seconds is plenty for most live conversations.

WebRTC (Voice SDK)

Send frames over the active call with sendAIConversationMessage. The SDK forwards the item as-is; in TypeScript its published AIConversationOutboundItem type does not list buffered image items yet, so cast the item until it does:

const canvas = document.createElement('canvas');
canvas.width = 640;
canvas.height = 480;
const context = canvas.getContext('2d');
if (!context) throw new Error('2D canvas is unavailable');

setInterval(() => {
  context.drawImage(videoElement, 0, 0, canvas.width, canvas.height);

  call.sendAIConversationMessage({
    type: 'message',
    role: 'user',
    buffer: true,
    captured_at: new Date().toISOString(),
    content: [{ type: 'input_image', image_url: canvas.toDataURL('image/jpeg', 0.7) }],
  } as any); // until the SDK type includes buffered image items
}, 2000);

Realtime WebSocket

On a realtime conversation socket, send the conversation.item.create frame shown above after session.created. For messages within the size limit, an invalid buffered item is answered with an invalid_frame error; the socket stays open.

Assistant event stream

On the customer-configured conversation event stream socket, send the same conversation.item.create frame. An invalid buffered item is answered with an error frame. Messages exceeding the 1 MiB transport limit also close the connection.

Privacy

The following describes the buffered-frame path. It does not describe storage or retention of MMS attachments or chat-widget image messages. The vision tool buffers image frames in memory for the running conversation and discards the buffer when it ends. Buffered images are not added to conversation history, transcripts, or webhooks, and reach the model only when the assistant invokes the tool. This does not disable or change recording. Separate video recordings — including recordings made through Telnyx Meeting — can retain the visual content shown during the conversation. Those recordings are stored and managed separately from the vision buffer. Telnyx Meeting sessions record automatically; see Collect Results for recording retrieval and deletion.

Guides

Missions

AI Insights

Assistants API

Missions API

Conversations API

For AI Agents

Vision tool for live conversations

How buffered frames work

Enable the vision tool

Send frames

WebRTC (Voice SDK)

Realtime WebSocket

Assistant event stream

Privacy

​Vision tool for live conversations

​How buffered frames work

​Enable the vision tool

​Send frames

​WebRTC (Voice SDK)

​Realtime WebSocket

​Assistant event stream

​Privacy

Vision tool for live conversations

How buffered frames work

Enable the vision tool

Send frames

WebRTC (Voice SDK)

Realtime WebSocket

Assistant event stream

Privacy