AI Gateway Inference API

The inference plane serves three endpoints. All of them authenticate with an AI Gateway token key (ltg_sk_...) issued through the management API. Telnyx account API keys, provider secrets and any other credential are rejected on this plane.

Endpoint	Purpose
`GET https://llm.telnyx.com/v1/models`	Models available to the supplied token key.
`POST https://llm.telnyx.com/v1/chat/completions`	OpenAI-compatible Chat Completions, including streaming. Works with every model.
`POST https://llm.telnyx.com/v1/messages`	Anthropic-compatible Messages, including streaming. Only for Anthropic BYOK models.

Compatibility is bounded by the AI Gateway contract, not by every option the OpenAI and Anthropic SDKs expose; see Supported request fields and Supported Messages fields.

Models

These Telnyx-hosted models are billed to your Telnyx account. Use these model names in the model field:

Kimi-K3
Kimi-K2.6
Kimi-K2.5
GLM-5.3
GLM-5.3-Flash
GLM-5.2
GLM-5.1-FP8
MiniMax-M3-MXFP8
MiniMax-M2.7
Qwen3.8-27B
Qwen3-235B-A22B
DeepSeek-V4.1-Flash
DeepSeek-V4-Flash-0731
Llama-3.3-70B-Instruct
Meta-Llama-3.1-70B-Instruct
Meta-Llama-3.1-8B-Instruct
gemma-2b-it

A group lists the model names its keys may call in allowed_models; a key can narrow that list further. Per-request output and prompt limits are listed under Request size limits.

Bring-your-own-key models

These models run on your own provider account with a provider key and are billed by that provider.

Provider	Models
Anthropic	`claude-fable-5-1`, `claude-opus-5-5`, `claude-sonnet-5`
OpenAI	`gpt-6-astra`, `gpt-6-sol`, `gpt-6-luna`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano`, `gpt-4.1`, `gpt-4.1-mini`, `gpt-4o`, `gpt-4o-mini`, `o3`

Add them to allowed_models like any other model. A BYOK model works only when the calling key’s group has an attached provider key for that model’s provider. Without one, requests fail with 503 and code enforcement_unavailable; they never fall back to a Telnyx-hosted model. Every BYOK model works on /v1/chat/completions. Anthropic BYOK models also work on /v1/messages.

Model discovery

GET /v1/models returns the models the supplied key may use: the intersection of the key’s allowed_models and its group’s list. It is not a global catalog, and it does not charge against any budget.

curl https://llm.telnyx.com/v1/models \
  -H "Authorization: Bearer $AI_GATEWAY_TOKEN_KEY"

{
  "object": "list",
  "data": [
    { "id": "Kimi-K3", "object": "model", "created": 1758535200, "owned_by": "telnyx" }
  ]
}

OpenAI SDK

Point the OpenAI SDK at the inference base URL and use the token key as the API key.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
    base_url="https://llm.telnyx.com/v1",
    max_retries=0,
)

completion = client.chat.completions.create(
    model=os.environ["AI_GATEWAY_MODEL"],
    messages=[{"role": "user", "content": "Tell me about Telnyx"}],
    max_tokens=256,
    user="application-bound-user-id",
)
print(completion.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_TOKEN_KEY,
  baseURL: "https://llm.telnyx.com/v1",
  maxRetries: 0,
});

const completion = await client.chat.completions.create({
  model: process.env.AI_GATEWAY_MODEL,
  messages: [{ role: "user", content: "Tell me about Telnyx" }],
  max_tokens: 256,
  user: "application-bound-user-id",
});
console.log(completion.choices[0].message.content);

curl -X POST https://llm.telnyx.com/v1/chat/completions \
  -H "Authorization: Bearer $AI_GATEWAY_TOKEN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kimi-K3",
    "messages": [{"role": "user", "content": "Tell me about Telnyx"}],
    "max_tokens": 256,
    "user": "application-bound-user-id"
  }'

The optional user field attributes the request to an application end user for budgeting and reporting. See End-user identity.

Streaming

Set stream: true and consume the stream to completion. A stream that starts with HTTP 200 can still fail later; handle SDK exceptions and close the stream.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
    base_url="https://llm.telnyx.com/v1",
    max_retries=0,
)

with client.chat.completions.create(
    model=os.environ["AI_GATEWAY_MODEL"],
    messages=[{"role": "user", "content": "Tell me about Telnyx"}],
    max_tokens=256,
    stream=True,
) as stream:
    for chunk in stream:
        if chunk.choices:
            print(chunk.choices[0].delta.content or "", end="", flush=True)

const stream = await client.chat.completions.create({
  model: process.env.AI_GATEWAY_MODEL,
  messages: [{ role: "user", content: "Tell me about Telnyx" }],
  max_tokens: 256,
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

Anthropic SDK

POST /v1/messages is available only for Anthropic models used with your own Anthropic key (Anthropic BYOK models). A request to /v1/messages with a Telnyx-hosted model or an OpenAI BYOK model returns 400 with code invalid_request; call those models on /v1/chat/completions. Anthropic BYOK models work on both endpoints.

The Anthropic SDK appends /v1/messages to its base URL, so set the base URL to the gateway root without the /v1 suffix. The SDK sends the token key in x-api-key, which the gateway accepts. Authorization: Bearer is also accepted; if both headers are present they must carry the same token.

import os
from anthropic import Anthropic

client = Anthropic(
    api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
    base_url="https://llm.telnyx.com",
    max_retries=0,
)

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=256,
    messages=[{"role": "user", "content": "Tell me about Telnyx"}],
    metadata={"user_id": "application-bound-user-id"},
)
print(message.content[0].text)

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  apiKey: process.env.AI_GATEWAY_TOKEN_KEY,
  baseURL: "https://llm.telnyx.com",
  maxRetries: 0,
});

const message = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 256,
  messages: [{ role: "user", content: "Tell me about Telnyx" }],
  metadata: { user_id: "application-bound-user-id" },
});
console.log(message.content[0].text);

curl -X POST https://llm.telnyx.com/v1/messages \
  -H "x-api-key: $AI_GATEWAY_TOKEN_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "max_tokens": 256,
    "messages": [{"role": "user", "content": "Tell me about Telnyx"}],
    "metadata": {"user_id": "application-bound-user-id"}
  }'

Raw HTTP requests must send anthropic-version: 2023-06-01; the SDKs add it automatically. metadata.user_id and the OpenAI user field identify the same end-user namespace. The endpoint streams standard Anthropic server-sent events (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop). An error after the stream has started arrives as an event: error frame.

with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=256,
    messages=[{"role": "user", "content": "Tell me about Telnyx"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

const stream = client.messages.stream({
  model: "claude-sonnet-5",
  max_tokens: 256,
  messages: [{ role: "user", content: "Tell me about Telnyx" }],
});

for await (const event of stream) {
  if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
    process.stdout.write(event.delta.text);
  }
}

Supported Messages fields

Field	Notes
`model`	Required. An Anthropic BYOK model available to this key.
`messages`	Required. 1 to 256 messages with roles `user` and `assistant`.
`max_tokens`	Required. At most 16,384.
`system`	String or array of `text` blocks.
`stream`	Anthropic server-sent events.
`temperature`	0 to 1.
`top_p`	0 to 1.
`top_k`	0 to 1000.
`stop_sequences`	Up to 4.
`metadata`	Only `user_id` is accepted; it maps to the end-user dimension.
`tools`, `tool_choice`	Tool definitions with `name`, `description` and `input_schema`. `tool_choice` accepts `auto`, `any`, `tool`, `none` or an object.

Content blocks of type text, tool_use and tool_result are supported. image blocks are rejected. OpenAI-specific fields (frequency_penalty, presence_penalty, n, seed, response_format, user, stream_options, parallel_tool_calls) are rejected on the Messages endpoint.

Retries and timeouts

Inference is not idempotent. An SDK retry after a timeout or disconnect can dispatch a second billed request, and a timeout is not evidence that the first request did no provider work. The examples on this page set max_retries=0; if your application retries, do so deliberately and only for errors that occurred before dispatch (for example 401, 403 or 429 with Retry-After). See Errors. Client-side timeouts are a client setting and do not extend the gateway’s server-side request lifetime.

Request size limits

Limit	Telnyx-hosted models	`gemma-2b-it`	BYOK models
Output tokens per request	8,192	2,048	16,384
Prompt size	About 32k tokens	About 8k tokens	About 128k tokens (about 96k for `gpt-4o` and `gpt-4o-mini`)

Prompt size is measured conservatively from the request size, so the usable limit can be somewhat lower than the model’s tokenizer would count. A request over either limit returns 400. Set max_tokens (or max_completion_tokens) on every request, especially when the key, user or group has a low tpm_limit. A request without it reserves the model’s full output allowance against budgets and tpm_limit, and can be rate-limited even when the actual response would be short. See Reservations.

Supported request fields

Unknown top-level fields are rejected with 400. Options that a specific model does not support are also rejected with 400 rather than silently ignored.

Chat Completions

Field	Notes
`model`	Required. Must be a model returned by `GET /v1/models` for this key.
`messages`	Required. Roles: `system`, `developer`, `user`, `assistant`, `tool`. Content is a string or an array of `text` and `image_url` parts.
`max_tokens` / `max_completion_tokens`	Send one or the other, never both. At most 8,192 for Telnyx-hosted models (2,048 for `gemma-2b-it`) and 16,384 for BYOK models. Strongly recommended; see Request size limits.
`stream`, `stream_options`	Server-sent events.
`temperature`	0 to 2.
`top_p`	0 to 1.
`stop`	String or up to 4 strings.
`seed`, `presence_penalty`, `frequency_penalty`, `n`	Passed through where the model supports them.
`user`	End-user identifier for budgeting and reporting.
`tools`, `tool_choice`, `parallel_tool_calls`	Function tools. `tool_choice` accepts `none`, `auto`, `required` or `{"type": "function", "function": {"name": ...}}`.
`response_format`	Structured output where the model supports it.

Image parts must be inline data:image/...;base64,... URLs (PNG, JPEG, GIF or WebP). Remote image URLs are rejected.

Not supported

Custom provider URLs, provider credentials, routing or fallback controls in the request body.
Remote image URLs.
Unlisted beta headers and options.
The Responses API, embeddings, image and audio generation, batches and realtime APIs. These are not part of the AI Gateway surface; see the Telnyx Inference API for those capabilities.

AI Gateway

API

Concepts

For AI Agents

AI Gateway Inference API

Models

Bring-your-own-key models

Model discovery

OpenAI SDK

Streaming

Anthropic SDK

Supported Messages fields

Retries and timeouts

Request size limits

Supported request fields

Chat Completions

Not supported

​Models

​Bring-your-own-key models

​Model discovery

​OpenAI SDK

​Streaming

​Anthropic SDK

​Supported Messages fields

​Retries and timeouts

​Request size limits

​Supported request fields

​Chat Completions

​Not supported

Models

Bring-your-own-key models

Model discovery

OpenAI SDK

Streaming

Anthropic SDK

Supported Messages fields

Retries and timeouts

Request size limits

Supported request fields

Chat Completions

Not supported