Talk to an AI Assistant over WebSocket

Client-to-Telnyx frame that supplies per-session dynamic variables. Optional for voice-only connections; required with avatar=true. Send it as the first frame, before streaming audio; the conversation starts as soon as it arrives for voice-only sessions, or after a short timeout with no client-supplied variables if it is never sent in voice-only mode. Avatar sessions do not start without this frame. Variables are nested under session.assistant.dynamic_variables. Only type is required: a bare frame with no session object is valid and starts voice-only conversations immediately with no client-supplied variables. With avatar=true, this frame begins avatar preparation; the conversation waits for avatar.media.ready after WebRTC negotiation and playback. Sending session.update after the session has started is rejected with a session_update_after_start error, and a dynamic_variables value that is not an object is rejected with invalid_session_update.

Client-to-Telnyx frame carrying a chunk of microphone audio. The audio field is a base64-encoded PCM16 payload at the sample rate requested with the input_sample_rate query parameter. Frames larger than 1 MiB are rejected with a frame_too_large error.

Client-to-Telnyx frame that adds a conversation item. Three item types are supported: a user message with input_text content (injects a completed user turn, which the assistant answers exactly as it would a spoken one); a buffered user message with "buffer": true and input_image content (stores image frames, such as webcam or screen-share captures, for the assistant's vision tool without triggering a response); and a function_call_output that returns the result of a client-side tool function_call previously requested by Telnyx. There is no separate frame to request a response — the assistant owns turn-taking. An item of any other shape is rejected with an invalid_item error, and a malformed buffered item with an invalid_frame error.

Client-to-Telnyx frame that cancels the in-progress assistant response (client-initiated barge-in without audio). Include response_id to target a specific response; omit it to cancel the current one. Cancelling flushes playback and Telnyx emits a response.done frame with status cancelled. Cancelling a stale or already-finished response is a no-op.

Client-to-Telnyx acknowledgment after applying the answer, establishing the peer connection, receiving live audio/video tracks and confirming the media receiver is ready. This acknowledgment starts the conversation and releases the greeting; session.created follows. Do not send microphone audio before session.created.

Client-to-Telnyx trickle ICE candidate. Send candidates after the offer; send candidate:null when gathering completes. Only the documented envelope and candidate fields are accepted. Server candidates are supplied in the SDP answer; there is no server ICE-candidate event.

Client-to-Telnyx SDP offer. Send exactly one offer after avatar.session.ready; audio and video transceivers must be recvonly. Renegotiation is not supported on the same connection.

Confirms the conversation has started and reports the negotiated audio formats. For voice-only connections this is the first session frame. With avatar=true it follows avatar signaling and the client avatar.media.ready acknowledgment.

Sent when server-side voice-activity detection determines the user has started speaking. Edge-triggered.

Sent when server-side voice-activity detection determines the user has stopped speaking. Edge-triggered, and never sent without a preceding speech-started.

Transcript of the audio the user streamed for the current turn.

Marks the start of an assistant turn. The response.id correlates the audio and transcript deltas that follow.

A chunk of the assistant''s synthesized speech, as base64-encoded PCM16 at the output rate announced in session.created.

A chunk of the text transcript of the assistant''s spoken response, streamed alongside the audio deltas.

Sent when the assistant has finished streaming audio for the turn.

Marks the end of an assistant turn, including its final status.

Telnyx-to-client frame requesting a client-side tool call. When the assistant invokes a client_side_tool, Telnyx sends a function_call item that the client executes locally. The client returns the result with a conversation.item.create frame carrying a function_call_output item that references the same call_id. The assistant continues once the result arrives, or after the tool times out.

Informational frame sent when the assistant invokes a server-side tool (for example a webhook or MCP integration). Telnyx executes tools server-side, so these frames are observational only: the client displays tool activity but does not execute or respond to the call.

Informational frame sent when a server-side tool invocation finishes, with its outcome.

Sent when an error occurs. Some errors are non-fatal and leave the session open; others close the connection after the frame is sent.

Avatar failure with code avatar_unavailable. Before session.created, error.fatal is true and Telnyx closes the WebSocket with code 1011; stop capture and close the peer connection. After session.created, error.fatal is false: the WebSocket remains open and the conversation continues audio-only. Stop WebRTC playback, close the peer connection, keep streaming microphone PCM and play subsequent response.output_audio.delta frames at the output format from session.created. Do not replay previously received audio. The reason is a sanitized cause identifier. Retryability and an optional delay describe retrying the avatar on a new connection, not renegotiating the current session. No provider credentials are exposed.

Telnyx-to-client SDP answer to the offer. The media receiver uses this remote description to establish synchronized audio and video reception.

Telnyx-to-client frame sent once the avatar is prepared, before session.created. The ice_configuration supplies ICE servers for the receive-only media connection. The audio field describes the avatar service audio format, not the microphone input rate.