Skip to main content

Gather using AI (BETA)

POST 
/calls/:call_control_id/actions/gather_using_ai

BETA Gather parameters defined in the request payload using a voice assistant.

You can pass parameters described as a JSON Schema object and the voice assistant will attempt to gather these informations. At the moment only English is supported.

Expected Webhooks (see callback schema below):

  • call.ai_gather.ended

Request ​

Path Parameters

    call_control_id stringrequired

    Unique identifier and token for controlling the call

Body

required

Gather using AI request

    parameters objectrequired

    The parameters described as a JSON Schema object that needs to be gathered by the voice assistant. See the JSON Schema reference for documentation about the format

    transcription

    object

    The settings associated with speech to text for the voice assistant. This is only relevant if the assistant uses a text-to-text language model. Any assistant using a model with native audio support (e.g. fixie-ai/ultravox-v0_4) will ignore this field.

    model string

    Default value: distil-whisper/distil-large-v2

    The speech to text model to be used by the voice assistant. distil-whisper/distil-large-v2 is lower latency but English-only. openai/whisper-large-v3-turbo is multi-lingual but slightly higher latency.

    voice VoiceConfig (string)

    Default value: Telnyx.LibriTTS.0

    The voice to be used by the voice assistant. Currently we support ElevenLabs, Telnyx and AWS voices.

    Supported Providers:

    • AWS: Use AWS.Polly.<VoiceId> (e.g., AWS.Polly.Joanna). For neural voices, which provide more realistic, human-like speech, append -Neural to the VoiceId (e.g., AWS.Polly.Joanna-Neural). Check the available voices for compatibility.

    • ElevenLabs: Use ElevenLabs.<ModelId>.<VoiceId> (e.g., ElevenLabs.BaseModel.John). The ModelId part is optional. To use ElevenLabs, you must provide your ElevenLabs API key as an integration secret under "voice_settings": {"api_key_ref": "<secret_id>"}. See integration secrets documentation for details. Check available voices.

    • Telnyx: Use Telnyx.<model_id>.<voice_id>

    voice_settings

    object

    The settings associated with the voice selected

    oneOf

    api_key_ref string

    The identifier for an integration secret /v2/integration_secrets that refers to your ElevenLabs API key. Warning: Free plans are unlikely to work with this integration.

    The settings associated with the voice selected

    greeting string

    Text that will be played when the gathering starts, if none then nothing will be played when the gathering starts. The greeting can be text for any voice or SSML for AWS.Polly.<voice_id> voices. There is a 3,000 character limit.

    assistant

    object

    model string

    Default value: meta-llama/Meta-Llama-3.1-70B-Instruct

    The model to be used by the voice assistant.

    instructions string

    The system instructions that the voice assistant uses during the gather command

    openai_api_key_ref string

    This is necessary only if the model selected is from OpenAI. You would pass the identifier for an integration secret /v2/integration_secrets that refers to your OpenAI API Key. Warning: Free plans are unlikely to work with this integration.

    message_history

    object[]

    The message history you want the voice assistant to be aware of, this can be useful to keep the context of the conversation, or to pass additional information to the voice assistant.

  • Array [

    • content string

    The content of the message

    role string

    Possible values: [assistant, user]

    The role of the message sender

  • ]

    • The message history you want the voice assistant to be aware of, this can be useful to keep the context of the conversation, or to pass additional information to the voice assistant.

    client_state string

    Use this field to add state to every subsequent webhook. It must be a valid Base-64 encoded string.

    command_id string

    Use this field to avoid duplicate commands. Telnyx will ignore any command with the same command_id for the same call_control_id.

    interruption_settings

    object

    Settings for handling user interruptions during assistant speech

    enable boolean

    Default value: true

    When true, allows users to interrupt the assistant while speaking

    Settings for handling user interruptions during assistant speech

Responses ​

200: Successful response upon making a call control command.

Schema

    data

    object

    result string

default: Unexpected error

Schema

    errors

    Error[]

  • Array [

    • code integerrequired
    title stringrequired
    detail string

    source

    object

    pointer json-pointer

    JSON pointer (RFC6901) to the offending entity.

    parameter string

    Indicates which query parameter caused the error.

    meta object

  • ]

Callbacks ​

Request samples

curl -L 'https://api.telnyx.com/v2/calls/:call_control_id/actions/gather_using_ai' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'Authorization: Bearer <TOKEN>' \
-d '{
  "parameters": {
    "properties": {
      "age": {
        "description": "The age of the customer.",
        "type": "integer"
      },
      "location": {
        "description": "The location of the customer.",
        "type": "string"
      }
    },
    "required": [
      "age",
      "location"
    ],
    "type": "object"
  },
  "voice": "Telnyx.LibriTTS.0",
  "voice_settings": {
    "voice_speed": 1.25
  },
  "greeting": "Hello, can you tell me your age and where you live?",
  "message_history": [
    {
      "content": "Hello, what'\''s your name?",
      "role": "assistant"
    },
    {
      "content": "Hello, I'\''m John.",
      "role": "user"
    }
  ],
  "client_state": "aGF2ZSBhIG5pY2UgZGF5ID1d",
  "command_id": "891510ac-f3e4-11e8-af5b-de00688a4901"
}'

Response samples

{
  "data": {
    "result": "ok"
  }
}