> ## Documentation Index
> Fetch the complete documentation index at: https://developers.telnyx.com/llms.txt
> Use this file to discover all available pages before exploring further.
# AI Gateway Inference API
> Call models through AI Gateway with the official OpenAI and Anthropic SDKs: Telnyx-hosted and bring-your-own-key models, base URL, authentication, streaming, request limits and the supported request subset.
The inference plane serves three endpoints. All of them authenticate with an AI Gateway token key (`ltg_sk_...`) issued through the [management API](/docs/inference/ai-gateway/management-api). Telnyx account API keys, provider secrets and any other credential are rejected on this plane.
| Endpoint | Purpose |
| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `GET https://llm.telnyx.com/v1/models` | Models available to the supplied token key. |
| `POST https://llm.telnyx.com/v1/chat/completions` | OpenAI-compatible Chat Completions, including streaming. Works with every model. |
| `POST https://llm.telnyx.com/v1/messages` | Anthropic-compatible Messages, including streaming. Only for [Anthropic BYOK models](#anthropic-sdk). |
Compatibility is bounded by the AI Gateway contract, not by every option the OpenAI and Anthropic SDKs expose; see [Supported request fields](#supported-request-fields) and [Supported Messages fields](#supported-messages-fields).
## Models
These Telnyx-hosted models are billed to your Telnyx account. Use these model names in the `model` field:
* `Kimi-K3`
* `Kimi-K2.6`
* `Kimi-K2.5`
* `GLM-5.3`
* `GLM-5.3-Flash`
* `GLM-5.2`
* `GLM-5.1-FP8`
* `MiniMax-M3-MXFP8`
* `MiniMax-M2.7`
* `Qwen3.8-27B`
* `Qwen3-235B-A22B`
* `DeepSeek-V4.1-Flash`
* `DeepSeek-V4-Flash-0731`
* `Llama-3.3-70B-Instruct`
* `Meta-Llama-3.1-70B-Instruct`
* `Meta-Llama-3.1-8B-Instruct`
* `gemma-2b-it`
A group lists the model names its keys may call in `allowed_models`; a key can narrow that list further. Per-request output and prompt limits are listed under [Request size limits](#request-size-limits).
### Bring-your-own-key models
These models run on your own provider account with a [provider key](/docs/inference/ai-gateway/byok) and are billed by that provider.
| Provider | Models |
| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Anthropic | `claude-fable-5-1`, `claude-opus-5-5`, `claude-sonnet-5` |
| OpenAI | `gpt-6-astra`, `gpt-6-sol`, `gpt-6-luna`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano`, `gpt-4.1`, `gpt-4.1-mini`, `gpt-4o`, `gpt-4o-mini`, `o3` |
Add them to `allowed_models` like any other model. A BYOK model works only when the calling key's group has an attached provider key for that model's provider. Without one, requests fail with `503` and code `enforcement_unavailable`; they never fall back to a Telnyx-hosted model.
Every BYOK model works on `/v1/chat/completions`. Anthropic BYOK models also work on `/v1/messages`.
### Model discovery
`GET /v1/models` returns the models the supplied key may use: the intersection of the key's `allowed_models` and its group's list. It is not a global catalog, and it does not charge against any budget.
```bash theme={null}
curl https://llm.telnyx.com/v1/models \
-H "Authorization: Bearer $AI_GATEWAY_TOKEN_KEY"
```
```json theme={null}
{
"object": "list",
"data": [
{ "id": "Kimi-K3", "object": "model", "created": 1758535200, "owned_by": "telnyx" }
]
}
```
## OpenAI SDK
Point the OpenAI SDK at the inference base URL and use the token key as the API key.
```python Python theme={null}
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
base_url="https://llm.telnyx.com/v1",
max_retries=0,
)
completion = client.chat.completions.create(
model=os.environ["AI_GATEWAY_MODEL"],
messages=[{"role": "user", "content": "Tell me about Telnyx"}],
max_tokens=256,
user="application-bound-user-id",
)
print(completion.choices[0].message.content)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const client = new OpenAI({
apiKey: process.env.AI_GATEWAY_TOKEN_KEY,
baseURL: "https://llm.telnyx.com/v1",
maxRetries: 0,
});
const completion = await client.chat.completions.create({
model: process.env.AI_GATEWAY_MODEL,
messages: [{ role: "user", content: "Tell me about Telnyx" }],
max_tokens: 256,
user: "application-bound-user-id",
});
console.log(completion.choices[0].message.content);
```
```bash curl theme={null}
curl -X POST https://llm.telnyx.com/v1/chat/completions \
-H "Authorization: Bearer $AI_GATEWAY_TOKEN_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "Kimi-K3",
"messages": [{"role": "user", "content": "Tell me about Telnyx"}],
"max_tokens": 256,
"user": "application-bound-user-id"
}'
```
The optional `user` field attributes the request to an application end user for budgeting and reporting. See [End-user identity](/docs/inference/ai-gateway/controls#end-user-identity).
### Streaming
Set `stream: true` and consume the stream to completion. A stream that starts with HTTP 200 can still fail later; handle SDK exceptions and close the stream.
```python Python theme={null}
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
base_url="https://llm.telnyx.com/v1",
max_retries=0,
)
with client.chat.completions.create(
model=os.environ["AI_GATEWAY_MODEL"],
messages=[{"role": "user", "content": "Tell me about Telnyx"}],
max_tokens=256,
stream=True,
) as stream:
for chunk in stream:
if chunk.choices:
print(chunk.choices[0].delta.content or "", end="", flush=True)
```
```javascript JavaScript theme={null}
const stream = await client.chat.completions.create({
model: process.env.AI_GATEWAY_MODEL,
messages: [{ role: "user", content: "Tell me about Telnyx" }],
max_tokens: 256,
stream: true,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
```
## Anthropic SDK
`POST /v1/messages` is available **only for Anthropic models used with your own Anthropic key** ([Anthropic BYOK models](#bring-your-own-key-models)). A request to `/v1/messages` with a Telnyx-hosted model or an OpenAI BYOK model returns `400` with code `invalid_request`; call those models on `/v1/chat/completions`. Anthropic BYOK models work on both endpoints.
The Anthropic SDK appends `/v1/messages` to its base URL, so set the base URL to the gateway root **without** the `/v1` suffix. The SDK sends the token key in `x-api-key`, which the gateway accepts. `Authorization: Bearer` is also accepted; if both headers are present they must carry the same token.
```python Python theme={null}
import os
from anthropic import Anthropic
client = Anthropic(
api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
base_url="https://llm.telnyx.com",
max_retries=0,
)
message = client.messages.create(
model="claude-sonnet-5",
max_tokens=256,
messages=[{"role": "user", "content": "Tell me about Telnyx"}],
metadata={"user_id": "application-bound-user-id"},
)
print(message.content[0].text)
```
```javascript JavaScript theme={null}
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
apiKey: process.env.AI_GATEWAY_TOKEN_KEY,
baseURL: "https://llm.telnyx.com",
maxRetries: 0,
});
const message = await client.messages.create({
model: "claude-sonnet-5",
max_tokens: 256,
messages: [{ role: "user", content: "Tell me about Telnyx" }],
metadata: { user_id: "application-bound-user-id" },
});
console.log(message.content[0].text);
```
```bash curl theme={null}
curl -X POST https://llm.telnyx.com/v1/messages \
-H "x-api-key: $AI_GATEWAY_TOKEN_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet-5",
"max_tokens": 256,
"messages": [{"role": "user", "content": "Tell me about Telnyx"}],
"metadata": {"user_id": "application-bound-user-id"}
}'
```
Raw HTTP requests must send `anthropic-version: 2023-06-01`; the SDKs add it automatically. `metadata.user_id` and the OpenAI `user` field identify the same end-user namespace.
The endpoint streams standard Anthropic server-sent events (`message_start`, `content_block_start`, `content_block_delta`, `content_block_stop`, `message_delta`, `message_stop`). An error after the stream has started arrives as an `event: error` frame.
```python Python theme={null}
with client.messages.stream(
model="claude-sonnet-5",
max_tokens=256,
messages=[{"role": "user", "content": "Tell me about Telnyx"}],
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
```
```javascript JavaScript theme={null}
const stream = client.messages.stream({
model: "claude-sonnet-5",
max_tokens: 256,
messages: [{ role: "user", content: "Tell me about Telnyx" }],
});
for await (const event of stream) {
if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
process.stdout.write(event.delta.text);
}
}
```
### Supported Messages fields
| Field | Notes |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `model` | Required. An Anthropic BYOK model available to this key. |
| `messages` | Required. 1 to 256 messages with roles `user` and `assistant`. |
| `max_tokens` | Required. At most 16,384. |
| `system` | String or array of `text` blocks. |
| `stream` | Anthropic server-sent events. |
| `temperature` | 0 to 1. |
| `top_p` | 0 to 1. |
| `top_k` | 0 to 1000. |
| `stop_sequences` | Up to 4. |
| `metadata` | Only `user_id` is accepted; it maps to the end-user dimension. |
| `tools`, `tool_choice` | Tool definitions with `name`, `description` and `input_schema`. `tool_choice` accepts `auto`, `any`, `tool`, `none` or an object. |
Content blocks of type `text`, `tool_use` and `tool_result` are supported. `image` blocks are rejected.
OpenAI-specific fields (`frequency_penalty`, `presence_penalty`, `n`, `seed`, `response_format`, `user`, `stream_options`, `parallel_tool_calls`) are rejected on the Messages endpoint.
## Retries and timeouts
Inference is **not idempotent**. An SDK retry after a timeout or disconnect can dispatch a second billed request, and a timeout is not evidence that the first request did no provider work. The examples on this page set `max_retries=0`; if your application retries, do so deliberately and only for errors that occurred before dispatch (for example `401`, `403` or `429` with `Retry-After`). See [Errors](/docs/inference/ai-gateway/errors).
Client-side timeouts are a client setting and do not extend the gateway's server-side request lifetime.
## Request size limits
| Limit | Telnyx-hosted models | `gemma-2b-it` | BYOK models |
| ------------------------- | -------------------- | --------------- | ------------------------------------------------------------ |
| Output tokens per request | 8,192 | 2,048 | 16,384 |
| Prompt size | About 32k tokens | About 8k tokens | About 128k tokens (about 96k for `gpt-4o` and `gpt-4o-mini`) |
Prompt size is measured conservatively from the request size, so the usable limit can be somewhat lower than the model's tokenizer would count. A request over either limit returns `400`.
Set `max_tokens` (or `max_completion_tokens`) on every request, especially when the key, user or group has a low `tpm_limit`. A request without it reserves the model's full output allowance against budgets and `tpm_limit`, and can be rate-limited even when the actual response would be short. See [Reservations](/docs/inference/ai-gateway/controls#reservations).
## Supported request fields
Unknown top-level fields are rejected with `400`. Options that a specific model does not support are also rejected with `400` rather than silently ignored.
### Chat Completions
| Field | Notes |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `model` | Required. Must be a model returned by `GET /v1/models` for this key. |
| `messages` | Required. Roles: `system`, `developer`, `user`, `assistant`, `tool`. Content is a string or an array of `text` and `image_url` parts. |
| `max_tokens` / `max_completion_tokens` | Send one or the other, never both. At most 8,192 for Telnyx-hosted models (2,048 for `gemma-2b-it`) and 16,384 for BYOK models. Strongly recommended; see [Request size limits](#request-size-limits). |
| `stream`, `stream_options` | Server-sent events. |
| `temperature` | 0 to 2. |
| `top_p` | 0 to 1. |
| `stop` | String or up to 4 strings. |
| `seed`, `presence_penalty`, `frequency_penalty`, `n` | Passed through where the model supports them. |
| `user` | End-user identifier for budgeting and reporting. |
| `tools`, `tool_choice`, `parallel_tool_calls` | Function tools. `tool_choice` accepts `none`, `auto`, `required` or `{"type": "function", "function": {"name": ...}}`. |
| `response_format` | Structured output where the model supports it. |
Image parts must be inline `data:image/...;base64,...` URLs (PNG, JPEG, GIF or WebP). Remote image URLs are rejected.
### Not supported
* Custom provider URLs, provider credentials, routing or fallback controls in the request body.
* Remote image URLs.
* Unlisted beta headers and options.
* The Responses API, embeddings, image and audio generation, batches and realtime APIs. These are not part of the AI Gateway surface; see the [Telnyx Inference API](/docs/inference/getting-started) for those capabilities.