​

Telnyx AI: Assistants — Full Documentation

Complete page content for Assistants (AI section) of the Telnyx developer docs (https://developers.telnyx.com). This file: https://developers.telnyx.com/development/llms/ai-assistants-llms-full-txt.md · Root index: https://developers.telnyx.com/llms.txt

​

Guides

​

Voice Assistant

Source: https://developers.telnyx.com/docs/inference/ai-assistants/no-code-voice-assistant.md

​

Video Tutorial

​

Requirements

1. Configure your AI Assistant
2. Configure the voice settings

​

Configure your AI Assistant

You are an intelligent and concise voice assistant. This is a {{telnyx_conversation_channel}} happening on {{telnyx_current_time}}. The agent is at {{telnyx_agent_target}} and the user is at {{telnyx_end_user_target}}.

Hi {{first_name}}, this is Nyx, your friendly Telnyx Assistant! How can I help you today?

​

Configure the voice settings

• TTS (Text-to-Speech): Telnyx, AWS, Azure, Elevenlabs, Inworld
• STT (Speech-to-Text): Telnyx (whisper), Deepgram, Azure — see Transcription Settings for model details and configuration

1. Wait seconds sets your baseline. A customer service agent might use 0.3 seconds for snappy responses. An agent calling into an IVR system needs 1.5 seconds to account for slower robotic speech.
2. On punctuation seconds handles high-confidence endpoints. When the transcription ends with a period or question mark, the user likely finished their thought. Set this to 0.1 seconds for minimal delay.
3. On no punctuation seconds handles uncertainty. The user said “my order number is” and paused. They’re probably looking at their screen. Set this to 1.5 seconds so the agent doesn’t interrupt with “I didn’t catch that” while they’re reading the digits.
4. On number seconds handles digit sequences specifically. People read numbers slowly: “4… 7… 2… 9.” Each pause could trigger a response. Extending this to 1.0 seconds prevents the agent from cutting them off at “4… 7…”

curl -X POST https://api.telnyx.com/v2/ai/assistants/{assistant_id} \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "telephony_settings": {
      "noise_suppression": "krisp"
    }
  }'

​

(Optional) Assign a phone number

​

(Optional) Enable messaging

​

Test it out!

• from the portal
• via API/CLI
• in an automated workflow like Zapier using our TeXML Outbound Call action.

​

Outbound calls via API

curl --request POST \
  --url https://api.telnyx.com/v2/texml/ai_calls/<texml_app_id> \
  --header "Authorization: Bearer $TELNYX_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
      "From": "+13128675309",
      "To": "+15551234567",
      "AIAssistantId": "assistant-6207ab25-b185-478f-b2ef-85159e226727"
  }'

curl --request POST \
  --url https://api.telnyx.com/v2/texml/ai_calls/<texml_app_id> \
  --header "Authorization: Bearer $TELNYX_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
      "From": "+13128675309",
      "To": "+15551234567",
      "AIAssistantId": "assistant-6207ab25-b185-478f-b2ef-85159e226727",
      "MachineDetection": "Enable",
      "AsyncAmd": true,
      "DetectionMode": "Premium"
  }'

​

Review the conversation

​

MMS integration during voice calls

​

How it works

• Automatically detect incoming MMS messages.
• Access and analyze attached images using vision-capable Language Models (VLMs).
• Provide real-time responses based on visual content.
• Continue the voice conversation with enhanced context.

​

Use cases

​

Configuration

1. Messaging must be enabled - Ensure messaging is enabled for your Voice AI Agent to receive MMS during calls.

2. Vision-capable models required - Use one of the two supported vision models: Groq/llama-4-maverick-17b-128e-instruct or OpenAI/gpt-4o for image processing.
3. Image processing - The assistant can handle common image formats (JPEG, PNG, etc.).

​

Best practices

• Model selection: Choose one of the two supported vision models when configuring your assistant.

• Response timing: The assistant will process images and respond within the voice call flow.
• Image quality: Higher resolution images provide better analysis results.
• Context integration: The assistant seamlessly combines visual and conversational context.

​

Supported language models

​

Optional enhancements

​

Integrations

• Salesforce - CRM and customer service
• ServiceNow - IT service management
• Jira - Project and issue tracking
• HubSpot - Marketing and sales CRM
• Zendesk - Customer support
• Intercom - Customer messaging
• Github - Code hosting and version control
• Greenhouse - Applicant tracking

​

Additional built-in tools

​

Model Context Protocol (MCP) Servers

• For call control applications, the conversation ID is returned by the Start AI Assistant command
• If you have configured a dynamic variables webhook URL, the conversation ID will be sent in this request payload at the start of a voice conversation.

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "some_mcp_method",
  "params": {
    "_meta": {
      "progressToken": null,
      "telnyx_conversation_id": "123"
    }
  }
}

​

Knowledge Bases

​

Insights

​

Embeddable Widget

​

Programmatic Voice

​

Third-party integrations

​

Vapi integration

1. Create a Vapi API Key.
2. Reference the key in your Assistant voice configuration.

Create a Vapi API Key

Reference the key in your Assistant voice configuration

​

ElevenLabs integration

1. Create an ElevenLabs API Key
2. Reference the key in your Assistant voice configuration

Create an ElevenLabs API Key

Reference the key in your Assistant voice configuration

​

OpenAI integration

1. Create an OpenAI API Key
2. Configure the language model in your AI Assistant

Create an OpenAI API Key

Configure the language model in your AI Assistant

​

Realtime Conversations

Source: https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md

​

How a conversation flows

Client → Telnyx   (connect wss://.../v2/ai/assistants/{assistant_id}/conversation)
Client ← Telnyx   {"type":"session.created", ...}
Client → Telnyx   {"type":"input_audio_buffer.append","audio":"<base64 pcm16>"}
Client → Telnyx   {"type":"input_audio_buffer.append","audio":"<base64 pcm16>"}
Client ← Telnyx   {"type":"input_audio_buffer.speech_started"}
Client ← Telnyx   {"type":"input_audio_buffer.speech_stopped"}
Client ← Telnyx   {"type":"conversation.item.input_audio_transcription.completed","transcript":"..."}
Client ← Telnyx   {"type":"response.created","response":{"id":"resp_9c1f4a2b"}}
Client ← Telnyx   {"type":"response.output_audio.delta","delta":"<base64 pcm16>", ...}
Client ← Telnyx   {"type":"response.output_audio_transcript.delta","delta":"We are open ", ...}
Client ← Telnyx   {"type":"response.output_audio.done","response_id":"resp_9c1f4a2b"}
Client ← Telnyx   {"type":"response.done","response":{"id":"resp_9c1f4a2b","status":"completed"}}

1. Open a WebSocket connection, optionally setting the audio format through query parameters.
2. Telnyx authenticates the connection, starts a conversation with the requested assistant, and sends a session.created frame confirming the negotiated input and output audio formats.
3. Stream microphone audio as input_audio_buffer.append frames.
4. Telnyx runs server-side voice-activity detection (VAD) and sends input_audio_buffer.speech_started and input_audio_buffer.speech_stopped on speech edges, followed by the user’s transcript.
5. The assistant responds with a response.created frame, a stream of audio and transcript deltas, then response.output_audio.done and response.done.
6. Speaking while the assistant is talking triggers barge-in: Telnyx stops the response and starts a new turn.

​

Connect and authenticate

wss://api.telnyx.com/v2/ai/assistants/{assistant_id}/conversation

Authorization: Bearer YOUR_TELNYX_API_KEY

import WebSocket from "ws";

const ASSISTANT_ID = "assistant-0f4e8b2a";
const url = `wss://api.telnyx.com/v2/ai/assistants/${ASSISTANT_ID}/conversation?input_sample_rate=16000`;

const ws = new WebSocket(url, {
  headers: { Authorization: `Bearer ${process.env.TELNYX_API_KEY}` },
});

ws.on("open", () => console.log("Connected"));

import asyncio
import os
import websockets

ASSISTANT_ID = "assistant-0f4e8b2a"
URL = f"wss://api.telnyx.com/v2/ai/assistants/{ASSISTANT_ID}/conversation?input_sample_rate=16000"

async def main():
    headers = {"Authorization": f"Bearer {os.environ['TELNYX_API_KEY']}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        print("Connected")

asyncio.run(main())

​

Session lifecycle

{
  "type": "session.created",
  "session": {
    "conversation_id": "3E6F995F-85F7-4705-9741-53B116D28237",
    "assistant_id": "assistant-0f4e8b2a",
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 16000 },
        "turn_detection": { "type": "server_vad" }
      },
      "output": {
        "format": { "type": "audio/pcm", "rate": 24000 }
      }
    }
  }
}

​

Handling audio

​

Stream microphone audio

{ "type": "input_audio_buffer.append", "audio": "<base64-encoded PCM16>" }

• Stream at a real-time pace, in small chunks (for example 20–100 ms of audio per frame). The server enforces an ingress budget; sending far faster than real time fails with an ingress_budget_exceeded error.
• Keep frames under 1 MiB. Larger frames are rejected with a frame_too_large error.
• Wait for session.created before sending audio.

import fs from "node:fs";

const SAMPLE_RATE = 16000;
const CHUNK_MS = 100;
const CHUNK_BYTES = (SAMPLE_RATE * 2 * CHUNK_MS) / 1000; // 16-bit mono

// speech.raw is raw little-endian PCM16 at 16 kHz
const pcm = fs.readFileSync("speech.raw");
let offset = 0;

const timer = setInterval(() => {
  if (offset >= pcm.length) {
    clearInterval(timer);
    return;
  }
  const chunk = pcm.subarray(offset, offset + CHUNK_BYTES);
  offset += CHUNK_BYTES;
  ws.send(
    JSON.stringify({
      type: "input_audio_buffer.append",
      audio: chunk.toString("base64"),
    })
  );
}, CHUNK_MS);

import base64
import json

SAMPLE_RATE = 16000
CHUNK_MS = 100
CHUNK_BYTES = SAMPLE_RATE * 2 * CHUNK_MS // 1000  # 16-bit mono

async def stream_audio(ws):
    # speech.raw is raw little-endian PCM16 at 16 kHz
    with open("speech.raw", "rb") as f:
        while chunk := f.read(CHUNK_BYTES):
            await ws.send(json.dumps({
                "type": "input_audio_buffer.append",
                "audio": base64.b64encode(chunk).decode(),
            }))
            await asyncio.sleep(CHUNK_MS / 1000)  # real-time pacing

function floatTo16BitPCM(float32Array) {
  const buffer = new ArrayBuffer(float32Array.length * 2);
  const view = new DataView(buffer);
  for (let i = 0; i < float32Array.length; i++) {
    const s = Math.max(-1, Math.min(1, float32Array[i]));
    view.setInt16(i * 2, s < 0 ? s * 0x8000 : s * 0x7fff, true);
  }
  return Buffer.from(buffer);
}

​

Speech detection and user transcripts

{ "type": "input_audio_buffer.speech_started" }
{ "type": "input_audio_buffer.speech_stopped" }

{
  "type": "conversation.item.input_audio_transcription.completed",
  "transcript": "What are your business hours?"
}

​

Receive assistant audio

let outputRate = 24000;
const playback = [];

ws.on("message", async (raw) => {
  const event = JSON.parse(raw.toString());

  switch (event.type) {
    case "session.created":
      outputRate = event.session.audio.output.format.rate;
      break;

    case "conversation.item.input_audio_transcription.completed":
      console.log(`You said: ${event.transcript}`);
      break;

    case "response.output_audio.delta":
      // Feed to your audio output as it arrives for lowest latency
      playback.push(Buffer.from(event.delta, "base64"));
      break;

    case "response.output_audio_transcript.delta":
      process.stdout.write(event.delta);
      break;

    case "response.done":
      console.log(`\n[turn ${event.response.id}: ${event.response.status}]`);
      break;

    case "error":
      console.error(`${event.error.code}: ${event.error.message}`);
      break;
  }
});

import base64
import json

async def handle_events(ws):
    output_rate = 24000
    playback = bytearray()

    async for raw in ws:
        event = json.loads(raw)
        match event["type"]:
            case "session.created":
                output_rate = event["session"]["audio"]["output"]["format"]["rate"]

            case "conversation.item.input_audio_transcription.completed":
                print(f"You said: {event['transcript']}")

            case "response.output_audio.delta":
                # Feed to your audio output as it arrives for lowest latency
                playback.extend(base64.b64decode(event["delta"]))

            case "response.output_audio_transcript.delta":
                print(event["delta"], end="", flush=True)

            case "response.done":
                print(f"\n[turn {event['response']['id']}: {event['response']['status']}]")

            case "error":
                print(f"{event['error']['code']}: {event['error']['message']}")

​

Interruption and barge-in

case "input_audio_buffer.speech_started":
  playback.length = 0; // drop queued assistant audio immediately
  break;

{ "type": "response.cancel" }

​

Send text instead of audio

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [{ "type": "input_text", "text": "What are your business hours?" }]
  }
}

​

Tool calls

​

Server-side tools: observe

{
  "type": "response.tool_call.started",
  "tool_call": { "id": "call_7a3f21b8", "name": "get_business_hours", "arguments": "{\"location\":\"downtown\"}" }
}

{
  "type": "response.tool_call.completed",
  "tool_call": { "id": "call_7a3f21b8", "name": "get_business_hours" },
  "status": "success"
}

​

Client-side tools: execute and respond

case "conversation.item.created": {
  const item = event.item;
  if (item.type === "function_call") {
    const args = JSON.parse(item.arguments);
    const result = await runTool(item.name, args); // your implementation
    ws.send(
      JSON.stringify({
        type: "conversation.item.create",
        item: {
          type: "function_call_output",
          call_id: item.call_id,
          output: JSON.stringify(result),
        },
      })
    );
  }
  break;
}

case "conversation.item.created":
    item = event["item"]
    if item["type"] == "function_call":
        args = json.loads(item["arguments"])
        result = await run_tool(item["name"], args)  # your implementation
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {
                "type": "function_call_output",
                "call_id": item["call_id"],
                "output": json.dumps(result),
            },
        }))

​

Error handling

{
  "type": "error",
  "error": { "code": "invalid_audio", "message": "Invalid base64-encoded audio payload" }
}

​

Learn more

• Migrate from the OpenAI Realtime API — Event-by-event mapping and behavioral differences
• Conversation WebSocket reference — The full frame-by-frame reference, under Assistants API → Conversation WebSocket in the sidebar
• Client-Side Tools — Configure tools that execute in your application
• Transcription Settings and Interruption Settings — Tune how the assistant hears and yields
• Voice Assistant Quickstart — Create and configure an assistant in the Portal

​

Migrate from OpenAI Realtime

Source: https://developers.telnyx.com/docs/inference/ai-assistants/migrate-from-openai-realtime.md

With OpenAI, the socket is the product: you configure the session, manage the conversation, and request every response. With Telnyx, the AI Assistant is the product: instructions, model, voice, and tools live on the assistant, and the assistant owns turn-taking. The socket only carries the conversation.

​

Before you start

​

Connection and authentication

import WebSocket from "ws";

const ws = new WebSocket("wss://api.openai.com/v1/realtime?model=gpt-realtime", {
  headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` },
});

ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.update",
    session: {
      type: "realtime",
      instructions: "You are a helpful assistant for Acme Inc.",
      audio: {
        input: { format: { type: "audio/pcm", rate: 24000 } },
        output: { voice: "marin" },
      },
    },
  }));
});

import WebSocket from "ws";

const ASSISTANT_ID = "assistant-0f4e8b2a"; // instructions, model, voice, tools already configured

const ws = new WebSocket(
  `wss://api.telnyx.com/v2/ai/assistants/${ASSISTANT_ID}/conversation?input_sample_rate=24000`,
  { headers: { Authorization: `Bearer ${process.env.TELNYX_API_KEY}` } }
);

// No session.update — wait for session.created and start streaming audio.

case "session.created":
  outputRate = event.session.audio.output.format.rate;
  break;

​

Event mapping

​

Client → server frames

​

Server → client frames

​

Turn-taking: delete your response orchestration

sendAudioChunks(ws, chunks);
ws.send(JSON.stringify({ type: "input_audio_buffer.commit" }));
ws.send(JSON.stringify({ type: "response.create" }));

sendAudioChunks(ws, chunks);
// That's it. Server VAD detects the end of speech and the assistant responds.

• Turn detection is always server_vad. You cannot disable it (turn_detection: null) or use semantic VAD. Push-to-talk UIs still work — only send audio while the button is held — but the turn boundary is still decided by server VAD, tuned via Interruption Settings.
• Automatic responses cannot be turned off. There is no create_response: false mode where you inspect the transcript before allowing a reply.
• Text turns need no trigger. conversation.item.create with input_text gets an automatic spoken response — do not follow it with response.create.

​

Interruption handling

case "input_audio_buffer.speech_started": {
  playback.length = 0;
  ws.send(JSON.stringify({
    type: "conversation.item.truncate",
    item_id: currentItemId,
    content_index: 0,
    audio_end_ms: msPlayedSoFar,
  }));
  break;
}

case "input_audio_buffer.speech_started":
  playback.length = 0; // just stop playing — Telnyx handles the rest
  break;

​

Function calling

• Tools your backend served (the common OpenAI pattern) usually become webhook or MCP tools: Telnyx calls your endpoint directly and the socket only shows informational response.tool_call.started / .completed frames. Your client-side function-calling code is deleted entirely.
• Tools that must run in your client process become client-side tools. The round-trip resembles OpenAI’s, with two changes: the request arrives as a single conversation.item.created frame with complete arguments (no function_call_arguments.delta streaming, no digging through response.done output), and you don’t send response.create after the output.

case "response.done": {
  for (const item of event.response.output ?? []) {
    if (item.type === "function_call") {
      const result = await runTool(item.name, JSON.parse(item.arguments));
      ws.send(JSON.stringify({
        type: "conversation.item.create",
        item: { type: "function_call_output", call_id: item.call_id, output: JSON.stringify(result) },
      }));
      ws.send(JSON.stringify({ type: "response.create" }));
    }
  }
  break;
}

case "conversation.item.created": {
  if (event.item.type === "function_call") {
    const result = await runTool(event.item.name, JSON.parse(event.item.arguments));
    ws.send(JSON.stringify({
      type: "conversation.item.create",
      item: { type: "function_call_output", call_id: event.item.call_id, output: JSON.stringify(result) },
    }));
    // No response.create — the assistant continues automatically.
  }
  break;
}

​

OpenAI features without a Telnyx equivalent

• Session reconfiguration — no session.update; configuration is fixed for the life of the connection.
• Manual turn control — no turn_detection: null, create_response: false, input_audio_buffer.commit/.clear, or semantic VAD.
• Out-of-band responses and custom context — no response.create, so no conversation: "none", per-response input arrays, response metadata, or per-response overrides (voice, modality, max_output_tokens).
• Text-only output — responses are always spoken; use the transcript deltas for text. (For a pure text channel, use the Assistants chat API instead — see Assistants API in the sidebar.)
• Conversation item manipulation — no conversation.item.truncate, .retrieve, or .delete, and no assistant-message injection into history.
• Image input — input_image content is not supported; conversation.item.create accepts input_text and function_call_output items only (anything else is rejected with invalid_item).
• G.711 audio — PCM16 only, in both directions.
• Rate-limit telemetry — no rate_limits.updated frames.

​

Migration checklist

1. Create an assistant and move session.update contents into its configuration (instructions, model, voice, tools, transcription, interruption).
2. Swap the URL and API key; pick input_sample_rate via query parameter.
3. Read the output sample rate from session.created instead of assuming 24 kHz.
4. Delete: session.update, response.create, input_audio_buffer.commit/.clear, conversation.item.truncate, and truncation bookkeeping.
5. Rewire function calling: backend tools → webhook/MCP tools (delete client code); in-client tools → handle conversation.item.created function_call items.
6. Keep: audio append loop, playback-flush on speech_started, response.cancel for programmatic interrupts, transcript rendering from .delta frames.
7. Update error handling to the Telnyx error codes, and treat a reconnect as a new conversation.
8. Test barge-in, tool calls, and long-silence behavior (session_idle_timeout) end to end.

​

Learn more

• Realtime voice conversations over WebSocket — The full guide to this API
• Conversation WebSocket reference — The complete frame-by-frame reference, under Assistants API → Conversation WebSocket in the sidebar
• Client-Side Tools — Tool handlers that run in your application
• Custom LLM — Bring your own model to a Telnyx assistant

​

Multi-Participant Calls

Source: https://developers.telnyx.com/docs/inference/ai-assistants/multi-participant-calls.md

• Add an Invite tool so your assistant can invite another participant to the current call.
• Design assistant instructions for multi-participant conversations.
• Use the Skip Turn tool so the assistant can stay silent while people talk to each other.
• Review a multi-participant call in Conversation History.
• Configure the same behavior through the Portal or the Assistants API.

​

How multi-participant calls work

• Tell whether the main user or an invited participant is speaking.
• Respond to either participant when appropriate.
• Use its configured tools, memory, dynamic variables, and integrations.
• Stay silent when the participants are talking to each other.
• Resume when someone addresses the assistant again.

​

Requirements

• A phone number or SIP URI for the assistant.
• A phone number or SIP URI for each participant the assistant may invite.
• Any tools the assistant should use after the participant joins, such as a calendar, CRM, or booking integration.

​

Step 1: Add an Invite tool

1. In the Mission Control Portal, open your assistant.
2. Go to the assistant’s Tools section.
3. Add an Invite tool.
4. Configure the invite target with the participant’s phone number or SIP URI.
5. Save the assistant.

​

Step 2: Give the assistant multi-participant instructions

You are Amber, a voice assistant helping coordinate a multi-participant call.

The call may include the main user and one or more invited participants. Pay close attention to who is speaking and who they are addressing.

When the main user asks you to invite someone, use the Invite tool. After the participant joins, briefly confirm that they joined, then let the participants talk.

If the participants are talking to each other and are not addressing you, use the Skip Turn tool and remain silent.

If someone addresses you directly, respond normally. You can use any of your configured tools to help complete the task.

​

Step 3: Add a Skip Turn tool

• Stay silent while invited participants talk to the main user.
• Avoid responding to side conversations.
• Wait until someone directly addresses the assistant.
• Let humans confirm details with each other before the assistant takes action.

​

Step 4: Use other assistant tools during the call

1. The assistant checks availability.
2. The assistant proposes a time to both participants.
3. The main user asks the invited participant whether the time works.
4. The assistant stays silent while the invited participant answers.
5. The assistant books the meeting after the participants confirm.

​

Best practices

​

Be explicit about when to speak

If James and Enzo are speaking directly to each other, use Skip Turn and stay silent. Only respond when someone addresses you by name or asks you to take an action.

​

Review calls in Conversation History

​

Next steps

• Build your first assistant with the Voice Assistant Quickstart.
• Add reusable tools from the Tools Library.
• Use dynamic variables to personalize calls.
• Review production calls with Agent Observability.

​

Memory

Source: https://developers.telnyx.com/docs/inference/ai-assistants/memory.md

• Specify which conversations your AI Assistant has memory access to
• Configure this dynamically at the start of every conversation

​

Identifying the conversations to include

• Every conversation it had with any user
• Every conversation it had with this specific user
• Every conversation it had with users in a specific group in the past 10 days
• Or something else entirely…

​

Configuring memory with the Dynamic Variables Webhook

{
  "data" :{
   "record_type": "event",
   "id": "event_id",
   "event_type": "assistant.initialization",
   "occurred_at": "2025-04-07T10:00:00Z",
   "payload": {
    "telnyx_conversation_channel": "phone_call",
    "telnyx_agent_target": "+1234567890",
    "telnyx_end_user_target": "+1234567890",
    "telnyx_end_user_target_verified": false
   }
  }
}

{
  "dynamic_variables": {
    "full_name": "Rachel Thomas",
    "facility_name": "UCHealth",
    "facility_department": "Cardiology"
  },
  "memory": {
    "conversation_query": "metadata->telnyx_end_user_target=eq.+13128675309&limit=5&order=last_message_at.desc"
  }
}

{
  "dynamic_variables": {
    "full_name": "Rachel Thomas",
    "facility_name": "UCHealth",
    "facility_department": "Cardiology"
  },
  "memory": {
    "conversation_query": "metadata->telnyx_end_user_target=eq.+13128675309&limit=5&order=last_message_at.desc",
    "insight_query": "insight_ids=123,456""
  }
}

​

Custom Metadata

{
  "dynamic_variables": {
    "full_name": "Rachel Thomas",
    "facility_name": "UCHealth",
    "facility_department": "Cardiology"
  },
  "memory": {
    "conversation_query": "metadata->telnyx_end_user_target=eq.+13128675309&limit=5&order=last_message_at.desc"
  },
  "conversation": {
    "metadata": {
      "your_custom_metadata": "your_custom_value"
    }
  }
}

​

Dynamic Variables

Source: https://developers.telnyx.com/docs/inference/ai-assistants/dynamic-variables.md

• Template your AI assistant configuration with dynamic variables.
• Supply the values via an outbound API request or the dynamic variable webhook.

​

Overview

​

How dynamic variables work

1. Define: Create placeholders in your assistant’s instructions, greeting, or tools using {{variable_name}} syntax.
2. Inject: Provide values through API calls, webhooks, SIP headers, or default configurations.
3. Resolve: Telnyx replaces placeholders with actual values when the conversation starts.
4. Use: Your assistant uses the personalized content throughout the conversation.

​

Key benefits

• Scalability: One assistant configuration serves multiple use cases.
• Personalization: Each conversation can be tailored to specific users or contexts.
• Efficiency: Reduce configuration overhead and maintenance complexity.
• Flexibility: Update values dynamically without modifying the assistant.

​

Common use cases

• Customer Service: Personalize greetings with customer names and account details.
• Appointment Scheduling: Include facility names, departments, and contact information.
• Account Management: Reference specific account numbers, balances, or service details.
• Healthcare: Customize interactions with patient names, appointment types, and provider information.

​

Best practices

​

Variable naming and organization

• Use descriptive names: Choose clear, meaningful variable names like customer_name instead of generic names like name.
• Follow consistent conventions: Use snake_case for variable names (facility_name, account_number).
• Group related variables: Organize logically related variables together (facility_name, facility_department, facility_contact).
• Avoid reserved namespaces: Don’t use the telnyx_ prefix, which is reserved for system variables.

// Good variable naming
{
  "customer_name": "Sarah Johnson",
  "account_number": "ACC-12345",
  "facility_name": "Memorial Hospital",
  "facility_department": "Cardiology"
}

// Poor variable naming
{
  "name": "Sarah",
  "num": "12345",
  "place": "Hospital",
  "dept": "Card"
}

​

Implementation strategy

• Start with defaults: Set default values in the Assistant builder for testing and as fallback values.
• Use API injection for dynamic data: Pass call-specific values through the AIAssistantDynamicVariables parameter in API calls.
• Leverage webhooks for complex lookups: Use the dynamic variables webhook for real-time data retrieval or database lookups.
• Implement graceful degradation: Ensure your assistant can handle scenarios where variables fail to resolve.

​

Security and data handling

• Protect sensitive information: Be cautious about including sensitive data in variables that may appear in logs.
• Validate webhook data: Implement proper validation for data returned from webhook endpoints.
• Consider data retention: Review your data retention and privacy requirements for variable content.
• Handle timeouts gracefully: Webhook responses must return within the configured timeout (default 1.5 seconds, up to 10 seconds via dynamic_variables_webhook_timeout_ms) or the call proceeds with fallback values.

​

Performance and reliability

• Optimize webhook response time: Keep webhook responses fast (well under your configured timeout; the default is 1.5 seconds).
• Implement error handling: Plan for scenarios where webhooks fail or return invalid data.
• Use caching strategies: Cache frequently accessed data to improve response times.
• Test across channels: Verify variable resolution works consistently across phone calls, web calls, and SMS.

​

Testing and debugging

• Test with Portal defaults: Use the Assistant builder’s default values to test variable behavior during development.
• Monitor webhook logs: Use the conversation transcript and webhook logs in the Portal to debug variable resolution issues.
• Validate fallback behavior: Ensure unresolved variables (displayed as {{variable_name}}) don’t break conversation flow.
• Test transfer scenarios: Verify that variables pass correctly when using transfer tools with SIP headers.

​

Cross-channel considerations

• Ensure universal compatibility: Test that your variables work across phone calls, web calls, and SMS channels.
• Handle channel-specific data: Consider how variables might behave differently across conversation channels.
• Plan for transfer scenarios: When transferring calls, use SIP headers to pass variable data to the receiving assistant.

​

Dynamic variables syntax

Hello, this is ABC Ambulance. Am I speaking with {{full_name}} with {{facility_name}} - {{facility_department}}?"

• Instructions
• Greeting
• Tools

​

Telnyx system variables

​

Date and time variables

​

Timezone variants

• {{telnyx_current_time_America/New_York}} → Monday, February 24 2025 11:04:15 AM EST
• {{telnyx_current_time_Australia/Sydney}} → Tuesday, February 25 2025 03:04:15 AM AEDT
• {{telnyx_current_time_of_day_Europe/London}} → 04:04:15 PM GMT

​

Custom formatting with the date filter

Common format codes

​

Customer-defined dynamic variables

curl --request POST \
  --url https://api.telnyx.com/v2/texml/ai_calls/<texml_app_id> \
  --header "Authorization: Bearer $TELNYX_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
      "From": "+13128675309",
      "To": "+15551234567",
      "AIAssistantId": "assistant-6207ab25-b185-478f-b2ef-85159e226727",
      "AIAssistantDynamicVariables": {
        "full_name": "James Smith",
        "facility_name": "Cleveland Clinic Main Campus",
        "facility_department": "Emergency Department"
      }
  }'

{
  "data": {
    "record_type": "event",
    "id": "event_12345678-90ab-cdef-1234-567890abcdef",
    "event_type": "assistant.initialization",
    "occurred_at": "2025-04-07T10:00:00Z",
    "payload": {
      "telnyx_conversation_channel": "phone_call",
      "telnyx_agent_target": "+13128675309",
      "telnyx_end_user_target": "+15551234567",
      "telnyx_end_user_target_verified": false,
      "call_control_id": "v3:u5OAKGEPT3Dx8SZSSDRWEMdNH2OripQhO",
      "assistant_id": "assistant_12345678-90ab-cdef-1234-567890abcdef"
    }
  }
}

{
  "dynamic_variables": {
    "full_name": "Rachel Thomas",
    "facility_name": "UCHealth",
    "facility_department": "Cardiology",
    "account_number": "ACC-12345",
    "appointment_time": "2:30 PM"
  },
  "memory": {
    "conversation_query": "metadata->telnyx_end_user_target=eq.+13128675309&limit=5&order=last_message_at.desc"
  },
  "conversation": {
    "metadata": {
      "customer_tier": "premium",
      "preferred_language": "en",
      "timezone": "America/Denver"
    }
  }
}

​

Troubleshooting

​

Common issues

​

Variables not resolving

• Webhook timeout: Your webhook endpoint took longer than the configured timeout (default 1.5 seconds) to respond, causing fallback to defaults.
• Variable name mismatch: Typos or case sensitivity differences between variable definition and usage.
• Incorrect precedence: Understanding the resolution order (API > SIP headers > webhook > defaults).
• Missing configuration: Variable not defined in any resolution method.

​

Incorrect formatting

• Invalid JSON: Webhook responses contain malformed JSON syntax.
• Wrong data types: Sending numbers instead of strings or vice versa.
• Missing required fields: Webhook response missing dynamic_variables object.
• Template syntax errors: Incorrect mustache template usage ({variable} instead of {{variable}}).

​

Timeout issues

• Slow database queries: Optimize database lookups in webhook endpoints.
• Network connectivity: Check firewall rules and network connectivity to webhook URLs.
• External service delays: Implement proper timeout handling for third-party API calls.
• Processing overhead: Minimize computation time in webhook response generation.

​

Debugging checklist

​

1. Verify variable definition

• Check variable name spelling and case sensitivity.
• Confirm variables are used in supported fields (instructions, greeting, tools).
• Validate mustache syntax: {{variable_name}}.
• Ensure variable names don’t use reserved telnyx_ prefix.

​

2. Check resolution priority

# Test API injection (highest priority)
curl --request POST \
  --url https://api.telnyx.com/v2/texml/ai_calls/<texml_app_id> \
  --header "Authorization: Bearer $TELNYX_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
      "From": "+13128675309",
      "To": "+15551234567",
      "AIAssistantId": "assistant-6207ab25-b185-478f-b2ef-85159e226727",
      "AIAssistantDynamicVariables": {
        "test_variable": "API_VALUE"
      }
  }'

• Verify SIP header format for transfer scenarios: X-Variable-Name becomes {{variable_name}}.
• Confirm webhook endpoint is reachable and responding correctly.
• Set default values in Assistant builder as fallback.

​

3. Validate webhook implementation

• Test webhook endpoint independently using tools like Postman.
• Verify JSON response format matches expected structure.
• Check response time (must be under the configured timeout; default is 1.5 seconds).
• Validate all required payload fields are present.

​

4. Test in Portal

• Use Assistant builder default values for initial testing.
• Review conversation logs for variable resolution behavior.
• Check webhook logs for errors, timeouts, or unexpected responses.

​

Webhook debugging

​

Using webhook logs effectively

• Request logs: Show the exact payload Telnyx sent to your webhook
• Response logs: Display your webhook’s response and any errors
• Timing information: Reveals if responses exceeded the configured timeout
• Error details: Provide specific error messages for failed requests

​

Common webhook issues

// Webhook response indicating auth failure
{
  "error": "Unauthorized: Invalid API key",
  "dynamic_variables": {}
}

// Incorrect - missing quotes around keys
{
  dynamic_variables: {
    customer_name: "John Doe"
  }
}

// Correct JSON format
{
  "dynamic_variables": {
    "customer_name": "John Doe"
  }
}

// Webhook response for timeout scenarios
{
  "dynamic_variables": {
    "customer_name": "Default Name"
  },
  "message": "External service timeout, using cached data"
}

​

Testing webhook endpoints

# Test your webhook with the exact payload Telnyx sends
curl --request POST \
  --url https://your-webhook-url.com/dynamic-variables \
  --header 'Content-Type: application/json' \
  --data '{
    "data": {
      "record_type": "event",
      "id": "event_12345678-90ab-cdef-1234-567890abcdef",
      "event_type": "assistant.initialization",
      "occurred_at": "2025-04-07T10:00:00Z",
      "payload": {
        "telnyx_conversation_channel": "phone_call",
        "telnyx_agent_target": "+13128675309",
        "telnyx_end_user_target": "+15551234567",
        "call_control_id": "v3:test-call-control-id"
      }
    }
  }'

​

Variable validation

​

Input validation

// Good - Clean, safe variable values
{
  "dynamic_variables": {
    "customer_name": "John Smith",
    "account_number": "ACC-12345",
    "appointment_time": "2:30 PM"
  }
}

// Avoid - Potentially problematic values
{
  "dynamic_variables": {
    "customer_name": "<script>alert('xss')</script>",
    "account_number": "'; DROP TABLE users; --",
    "appointment_time": null
  }
}

• Escape quotes and backslashes in variable content.
• Validate data types match expected formats.
• Provide meaningful defaults for empty or null values.

​

Response format validation

// Complete webhook response format
{
  "dynamic_variables": {
    "customer_name": "Sarah Johnson",
    "facility_name": "Memorial Hospital"
  },
  "memory": {
    "conversation_query": "metadata->customer_id=eq.12345"
  },
  "conversation": {
    "metadata": {
      "customer_tier": "premium"
    }
  }
}

​

Testing scenarios

• Verify variables work consistently across phone calls, web calls, and SMS.
• Test variable behavior in transfer scenarios using SIP headers.
• Validate fallback behavior when webhook endpoints are unavailable.

• Test webhook endpoint performance under expected call volumes.
• Verify response times remain under your configured timeout (default 1.5 seconds) during peak usage.
• Monitor for memory leaks or performance degradation over time.

​

Conversation Workflows

Source: https://developers.telnyx.com/docs/inference/ai-assistants/workflows.md

​

Why use workflows?

• Break complex conversations into focused steps: Give each stage its own name and instructions, such as Intake, Billing, Schedule appointment, or Escalate to specialist.
• Route with natural language or deterministic rules: Move between steps when the LLM decides a condition is met, or when a dynamic variable matches a configured value.
• Tune behavior per step: Append or replace the assistant’s main instructions, and optionally override the model or voice for a specific node.
• Route to another assistant: Route from one workflow to another assistant when the conversation should be handled by a different specialist.
• Debug what happened later: Conversation transcripts can show the workflow node associated with assistant messages, so you can connect real conversations back to the workflow design.

​

How workflows work

• Nodes: Conversation steps. A node is either a prompt node (an LLM-driven step with its own label, instructions, instruction mode, model override, and voice override) or a speak node (a deterministic step that plays a fixed scripted message with no LLM turn).
• Edges: Transitions between nodes, or from a node to another assistant. Each edge has a condition that decides when that path should be taken. Conditions can be natural-language (LLM), deterministic (variable comparison), or a default fallback.

​

What you can build with workflows

• Start from a defined entry point: Choose which node begins the workflow.
• Create focused conversation stages: Add prompt nodes for each step and give each one its own instructions.
• Play scripted lines verbatim: Add speak nodes for greetings, disclosures, or compliance statements that must be delivered word-for-word, with no model turn.
• Control how prompts combine: Append node instructions to the assistant’s base instructions, or replace the base instructions for a specific step.
• Tune individual steps: Override the model, voice, tools, or transcription behavior for a node when that step needs different capabilities.
• Connect steps with conditional routing: Use LLM conditions for natural-language decisions and variable comparisons for deterministic decisions.
• React to tool outcomes: Route based on whether a configured tool succeeds or fails.
• Move across assistants: Send a conversation from one workflow into another assistant when a specialist configuration should take over.
• Review workflow context in transcripts: Use conversation history to understand which workflow step produced assistant messages.

​

Create a workflow in the Portal

1. Open AI Assistants in the Telnyx Portal.
2. Select an assistant.
3. Open the Workflow tab.
4. Enable workflows if the assistant does not already have one.
5. Add nodes for each major stage of the conversation.
6. Connect nodes with edges and configure the condition for each edge.
7. Save the assistant.

​

Configure workflow nodes

​

Node types

• Prompt node: An LLM-driven step. The assistant generates its response from the node’s instructions (combined with the assistant’s base instructions), using the node’s model, voice, and tool settings. Most workflow steps are prompt nodes. This is the default node type.
• Speak node: A deterministic step that plays a fixed, scripted message and then advances. The assistant does not call the model on a speak node, so the wording is exactly what you configure. Use speak nodes for greetings, disclosures, hold messages, compliance statements, or any line that must be delivered verbatim.

​

Speak node message

• The message is delivered verbatim — there is no model turn, so the caller hears exactly what you type.
• The message supports {{variable}} placeholders. Both system variables and custom dynamic variables defined on the assistant are interpolated at runtime, with inline highlighting and autocomplete in the editor.
• A speak node must always have exactly one outgoing default edge. Because a speak node does not make a routing decision itself, it advances along this default edge after delivering the message.

Thanks for calling Acme, {{first_name}}. This call may be recorded for quality and training.

​

Node name

• Greeting & Identify Intent
• Answer FAQ
• Transfer Call
• Collect appointment details

​

Node instructions

• Append to assistant instructions: Keep the assistant’s base behavior and add step-specific guidance.
• Replace assistant instructions: Use only the node’s instructions for this step.

​

Tool availability

• Inherited tools: Every tool configured on the assistant. Each tool has a toggle so you can enable or disable it for this specific node. Tools are enabled by default, so a new node starts with the full assistant toolset until you turn tools off.
• Added for this node: Tools attached to this node only. Use the Select a new tool dropdown to add a node-specific tool that should not be available elsewhere in the workflow.

​

Model override

• Use a faster model for intake.
• Use a stronger model for complex qualification or policy-heavy decisions.
• Keep the assistant default everywhere except one high-value decision point.

​

Voice override

• Use the standard brand voice during greeting and intent detection.
• Switch to a calmer voice for sensitive support flows.
• Use a different voice when routing to a specialist assistant persona.

​

Configure workflow edges

• Source node: The node the conversation is leaving.
• Target type: Another workflow node, or another assistant.
• Condition type: The logic that decides whether the edge should be followed — an LLM condition, a variable comparison, or a default fallback.

​

LLM conditions

The caller has a general office-hours, location, or FAQ question.

The caller wants to speak with a specific person or department.

The user has asked to speak with a human agent.

​

Variable comparison conditions

• telnyx_conversation_channel == "phone_call"
• customer_tier == "enterprise"
• telnyx_conversation_duration_secs >= 30
• telnyx_shaken_stir_attestation != "a"

​

Default conditions

• A speak node delivers a scripted message and then advances. It must have exactly one outgoing default edge so the conversation always has a defined next step.
• Default conditions are only valid on edges that leave a speak node. They are not used on edges leaving a prompt node, which route based on LLM or variable comparison conditions.

​

Configure workflows with the API

{
  "conversation_flow": {
    "start_node_id": "n_greeting",
    "nodes": [
      {
        "id": "n_greeting",
        "name": "Greeting & Identify Intent",
        "instructions": "Quickly determine whether the caller needs a person, a department, or a general FAQ answer.",
        "instructions_mode": "append"
      },
      {
        "id": "n_faq",
        "name": "Answer FAQ",
        "instructions": "Answer hours, location, and general information questions. If they need more detail, offer to connect them to the right person.",
        "instructions_mode": "append"
      },
      {
        "id": "n_transfer",
        "name": "Transfer Call",
        "instructions": "Confirm the destination department or person, then verbally confirm the transfer before connecting the caller.",
        "instructions_mode": "append"
      }
    ],
    "edges": [
      {
        "id": "e_greeting_to_faq",
        "start_node_id": "n_greeting",
        "target": { "type": "node", "node_id": "n_faq" },
        "condition": {
          "type": "llm",
          "prompt": "The caller has a general office-hours, location, or FAQ question."
        }
      },
      {
        "id": "e_greeting_to_transfer",
        "start_node_id": "n_greeting",
        "target": { "type": "node", "node_id": "n_transfer" },
        "condition": {
          "type": "llm",
          "prompt": "The caller wants to speak with a specific person or department."
        }
      }
    ]
  }
}

​

Speak nodes and default edges in the API

{
  "conversation_flow": {
    "start_node_id": "n_disclosure",
    "nodes": [
      {
        "type": "speak",
        "id": "n_disclosure",
        "name": "Recording Disclosure",
        "message": "Thanks for calling Acme, {{first_name}}. This call may be recorded for quality and training."
      },
      {
        "type": "prompt",
        "id": "n_intake",
        "name": "Identify Intent",
        "instructions": "Find out what the caller needs and route them accordingly.",
        "instructions_mode": "append"
      }
    ],
    "edges": [
      {
        "id": "e_disclosure_to_intake",
        "start_node_id": "n_disclosure",
        "target": { "type": "node", "node_id": "n_intake" },
        "condition": { "type": "default" }
      }
    ]
  }
}

​

Route to another assistant

• A sales assistant routing qualified technical questions to a solutions assistant.
• A front-desk assistant routing billing questions to a billing assistant.
• A general support assistant routing high-risk cases to a stricter compliance assistant.

​

Example workflow patterns

​

Front desk receptionist

Greeting & Identify Intent
├── Answer FAQ       when the caller asks about hours, location, or general information
└── Transfer Call    when the caller wants a person or department

​

Appointment booking

Collect request
→ Collect availability
→ Confirm details
→ Final confirmation

​

Escalation after timeout

Troubleshooting
├── Continue troubleshooting  when the issue is not resolved
└── Route to specialist       when telnyx_conversation_duration_secs >= 300

​

Multi-assistant specialization

Main support assistant
├── Billing assistant
├── Technical support assistant
└── Sales assistant

​

Test and debug workflows

• Check whether the assistant followed the expected path.
• Look for assistant messages labeled with workflow node context.
• Open the workflow from transcript context when you need to inspect the node that produced a response.
• Review dynamic variable values and webhook behavior if a variable comparison did not route as expected.

​

Best practices

​

Keep nodes focused

​

Limit each node to the tools it needs

​

Prefer append mode for global rules

​

Write edge conditions as clear decisions

​

Use variable comparisons for hard rules

​

Avoid too many paths from one node

​

Test every path before production

​

Related resources

• Dynamic Variables: Personalize prompts and route with runtime data.
• Version Testing and Traffic Distribution: Test assistant changes before sending all traffic to a new version.
• Integrations: Connect assistants to external systems and data sources.

​

Async Tools

Source: https://developers.telnyx.com/docs/inference/ai-assistants/async-tools.md

• How to configure webhook tools to run asynchronously
• How to use the Add Messages API to inject context mid-conversation
• How to combine both features for powerful async workflows

​

Overview

​

The two building blocks

​

Combined workflow

1. Assistant triggers an async webhook (e.g., order lookup)
2. Assistant continues chatting with the customer
3. Backend processes the request (5, 10, 30 seconds later)
4. Backend calls Add Messages API to inject the results
5. Assistant naturally incorporates the new information

​

Async webhooks

​

Configuring an async webhook

{
  "type": "webhook",
  "webhook": {
    "name": "lookup_order_status",
    "description": "Triggers an async order status lookup. Results will be delivered automatically when ready.",
    "url": "https://your-backend.com/order-lookup",
    "method": "POST",
    "async": true,
    "headers": [
      {"name": "Content-Type", "value": "application/json"}
    ],
    "body_parameters": {
      "type": "object",
      "properties": {
        "order_id": {
          "type": "string",
          "description": "The customer's order ID"
        }
      },
      "required": ["order_id"]
    }
  }
}