# Telnyx AI: AI Gateway — Full Documentation
> Complete page content for AI Gateway (AI section) of the Telnyx developer docs (https://developers.telnyx.com).
> This file: https://developers.telnyx.com/docs/development/llms/ai-ai-gateway-llms-full-txt · Root index: https://developers.telnyx.com/llms.txt

## AI Gateway

### Overview

> Source: https://developers.telnyx.com/docs/inference/ai-gateway.md

AI Gateway issues **scoped inference credentials** for applications. Instead of sharing a Telnyx account API key with every service, create a token group, issue a token key inside it and hand that key to the application. The gateway enforces the model allowlist, budget and rate limits attached to the key, and records every request in a usage ledger attributed to the key, its user, its group and the end user it served.

Applications call the gateway with the official OpenAI SDKs by changing the base URL and the API key. No request rewriting is required. Anthropic models used with your own Anthropic key can also be called with the official Anthropic SDKs.

## Capabilities

| Capability | Behavior |
| --- | --- |
| Token groups | Define model access, group budgets and rate limits. |
| Token users | Associate an application actor with one or more groups; aggregate its limits across all of its keys. |
| Token keys | Issue scoped inference credentials for a user or a service; narrow model access, set limits and expiry, revoke access. |
| End users | Apply account-scoped budget caps and blocks to a caller-asserted application user identifier. |
| Provider keys (BYOK) | Store an OpenAI or Anthropic key once and attach it to groups; your provider bills requests on bring-your-own-key models. |
| Inference | OpenAI Chat Completions, Anthropic Messages (Anthropic BYOK models), [model discovery](/docs/inference/ai-gateway/inference-api#models) and streaming. |
| Guardrails | [Flag or block supported secrets and sensitive data](/docs/inference/ai-gateway/guardrails) with a policy on each token group. |
| Usage | Durable request accounting with reservations, corrections and dimensional summaries. |

## Two planes, two credentials

The gateway exposes a **management plane** for provisioning and reporting, and an **inference plane** that applications call. They use different hostnames and different credentials.

| Plane | Base URL | Credential |
| --- | --- | --- |
| Management | `https://api.telnyx.com/v2/llm_token_gateway` | Telnyx account API key: `Authorization: Bearer $TELNYX_API_KEY` |
| OpenAI-compatible inference | `https://llm.telnyx.com/v1` | AI Gateway token key: `Authorization: Bearer $AI_GATEWAY_TOKEN_KEY` |
| Anthropic-compatible inference (Anthropic BYOK models) | `https://llm.telnyx.com` as the SDK base URL | Same token key via `x-api-key`; the SDK appends `/v1/messages` |

Token keys start with `ltg_sk_`. The inference plane rejects Telnyx account API keys and provider secrets; the management plane rejects token keys. Keep every credential in a trusted backend or secret store, never in browser code, source control or logs.

## How it works

1. **Create a token group** with an explicit `allowed_models` list and optional budget and rate limits. To use your own OpenAI or Anthropic account, attach a [provider key](/docs/inference/ai-gateway/byok) to the group.
2. **Issue a token key** in that group, optionally bound to a token user. The secret is returned once, on the create response.
3. **Call models** from the application with an OpenAI SDK (or, for Anthropic BYOK models, an Anthropic SDK) pointed at the inference base URL and authenticated with the token key.
4. **Inspect usage** with the spend events and spend summary endpoints, filtered by group, user, key or end user.
5. **Revoke** the key when the application no longer needs it. New admissions stop immediately; spend history is retained.

Telnyx owns authorization, admission, revocation and the authoritative usage ledger. Applications never hold provider credentials or configure model providers directly.

Usage of Telnyx-hosted models is billed to your Telnyx account at standard Telnyx AI Inference pricing for each model. Requests on bring-your-own-key models are billed by your provider on your provider account. Budgets and reported `cost` values use a flat reference rate for enforcement and attribution; see [Budgets](/docs/inference/ai-gateway/controls#budgets).

## Next steps

    Create a group, issue a token key, make a request and revoke the key.
    Telnyx-hosted and BYOK models, OpenAI and Anthropic SDK configuration, streaming, request limits and supported fields.
    Groups, users, keys, end users and provider keys, with idempotency and ETag rules.
    How each control is enforced and what happens at the limit.
    Group policies, prompt and response inspection, streaming and findings.
    Use your own OpenAI or Anthropic key; your provider bills those requests.
    Spend events, dimensional summaries and snapshot pagination.
    Status codes, structured error codes and how to handle them.

---

### Quickstart

> Source: https://developers.telnyx.com/docs/inference/ai-gateway/quickstart.md

This quickstart provisions a group and a service token key through the management API, makes an inference request with that key, reads the resulting usage and revokes the key. Every step is a copy-paste request.

## Prerequisites

- A Telnyx API key from the [portal](https://portal.telnyx.com/#/api-keys).
- A model name from the [available models](/docs/inference/ai-gateway/inference-api#models). This guide uses `Kimi-K3`. The inference `GET /v1/models` endpoint is scoped to a token key, so it cannot be used to discover models before a key exists.
- `curl`, `jq` and `uuidgen` (or another way to generate a UUID for the `Idempotency-Key` header).

Export the credential, the model name and the two base URLs so the examples work as-is:

```bash
export TELNYX_API_KEY="KEY..."
export AI_GATEWAY_MODEL="Kimi-K3"
export AI_GATEWAY_MANAGEMENT_BASE_URL="https://api.telnyx.com/v2/llm_token_gateway"
export AI_GATEWAY_INFERENCE_BASE_URL="https://llm.telnyx.com/v1"
```

  This walkthrough creates persistent account resources and can incur model charges.

A token group defines which models its keys may call and the limits they share. `name` and `allowed_models` are required. An empty `allowed_models` list permits no inference.

This example sets a USD 10 budget per anchored one-day period and 60 requests per minute.

```bash
curl -X POST "$AI_GATEWAY_MANAGEMENT_BASE_URL/token_groups" \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d "$(jq -n --arg model "$AI_GATEWAY_MODEL" '{
    name: "support-assistant",
    allowed_models: [$model],
    max_budget: 10,
    budget_duration: "1d",
    rpm_limit: 60
  }')"
```

```json
{
  "data": {
    "record_type": "token_group",
    "id": "5f1c9d2e-7b3a-4c8e-9f21-0a6d4e8b1c33",
    "name": "support-assistant",
    "allowed_models": ["Kimi-K3"],
    "max_budget": 10,
    "budget_duration": "1d",
    "rpm_limit": 60,
    "tpm_limit": null,
    "provider_key_ids": [],
    "cache_ttl": null,
    "guardrails": null,
    "fallbacks": null,
    "retries": null,
    "blocked": false,
    "spend": 0,
    "reserved_spend": 0,
    "budget_started_at": "2026-09-22T10:00:00Z",
    "resets_at": "2026-09-23T10:00:00Z",
    "version": 1,
    "created_at": "2026-09-22T10:00:00Z",
    "updated_at": "2026-09-22T10:00:00Z"
  }
}
```

Keep `data.id`; the next step needs it and the group is what you retire at the end.

```bash
export GROUP_ID="5f1c9d2e-7b3a-4c8e-9f21-0a6d4e8b1c33"
```

Every mutation requires an `Idempotency-Key`. If a request times out, retry it with the **same** key and body; a new key starts a different operation and can create a second resource.

A token key is the credential the application uses. `token_user_id: null` creates a **service key** owned by the group alone. Null `allowed_models` inherits the group's list. Key limits that are omitted are set to their maximums: USD 1,000 lifetime budget, 6,000 requests and 10,000,000 tokens per minute. See [Token key limits](/docs/inference/ai-gateway/controls#token-key-limits).

```bash
curl -X POST "$AI_GATEWAY_MANAGEMENT_BASE_URL/token_keys" \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d "$(jq -n --arg group "$GROUP_ID" '{
    name: "support-backend",
    token_group_id: $group,
    token_user_id: null
  }')"
```

```json
{
  "data": {
    "record_type": "token_key",
    "id": "9b7e4a10-3c2d-4f5e-8a6b-1d2c3e4f5a60",
    "name": "support-backend",
    "token_group_id": "5f1c9d2e-7b3a-4c8e-9f21-0a6d4e8b1c33",
    "token_user_id": null,
    "allowed_models": null,
    "max_budget": 1000,
    "budget_duration": null,
    "rpm_limit": 6000,
    "tpm_limit": 10000000,
    "expires_at": null,
    "blocked": false,
    "required_end_user_id": false,
    "spend": 0,
    "reserved_spend": 0,
    "budget_started_at": null,
    "resets_at": null,
    "version": 1,
    "created_at": "2026-09-22T10:01:00Z",
    "updated_at": "2026-09-22T10:01:00Z",
    "token": "ltg_sk_SYNTHETIC_EXAMPLE_NOT_A_VALID_CREDENTIAL"
  }
}
```

  The example token is synthetic. Use `data.token` from your create response; it is returned **only on the original create response**. GET, list and idempotent replays return metadata without the secret. If the response is lost, revoke the key and create a new one with a new idempotency key.

Store the token in your secret store now, then export it and the key ID for the remaining steps:

```bash
export TOKEN_KEY_ID="9b7e4a10-3c2d-4f5e-8a6b-1d2c3e4f5a60"
export AI_GATEWAY_TOKEN_KEY="ltg_sk_..."
```

To issue a key for a specific application user instead, first `POST /token_users` with `name` and `token_group_ids: [GROUP_ID]`, then pass the returned ID as `token_user_id`. Key ownership cannot be changed later; issue a replacement key instead.

The inference plane authenticates with the token key, not the account API key. `GET /v1/models` returns only the models this key may use.

```bash
curl "$AI_GATEWAY_INFERENCE_BASE_URL/models" \
  -H "Authorization: Bearer $AI_GATEWAY_TOKEN_KEY"
```

Send a Chat Completions request with one of those models:

```bash curl
curl -X POST "$AI_GATEWAY_INFERENCE_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_TOKEN_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg model "$AI_GATEWAY_MODEL" '{
    model: $model,
    messages: [{role: "user", content: "Hello"}],
    max_tokens: 32
  }')"
```

```python Python
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
    base_url=os.environ["AI_GATEWAY_INFERENCE_BASE_URL"],
    max_retries=0,
)

completion = client.chat.completions.create(
    model=os.environ["AI_GATEWAY_MODEL"],
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=32,
)
print(completion.choices[0].message.content)
```

```javascript JavaScript
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_TOKEN_KEY,
  baseURL: process.env.AI_GATEWAY_INFERENCE_BASE_URL,
  maxRetries: 0,
});

const completion = await client.chat.completions.create({
  model: process.env.AI_GATEWAY_MODEL,
  messages: [{ role: "user", content: "Hello" }],
  max_tokens: 32,
});
console.log(completion.choices[0].message.content);
```

Inference requests are **not idempotent**. The examples disable SDK retries so that a timeout cannot silently trigger a second billed request. Set `max_tokens` on every request: a request without it reserves the model's full output allowance against budgets and `tpm_limit`. See [Inference API](/docs/inference/ai-gateway/inference-api) for streaming and request size limits.

Spend events are queried over a half-open UTC date range `[start_date, end_date)` of at most 31 days, using ISO dates rather than timestamps. Filter by the key you just used:

```bash
curl --globoff -G "$AI_GATEWAY_MANAGEMENT_BASE_URL/spend/events" \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  --data-urlencode "start_date=2026-09-22" \
  --data-urlencode "end_date=2026-09-23" \
  --data-urlencode "token_key_id=$TOKEN_KEY_ID" \
  --data-urlencode "page[size]=100"
```

```json
{
  "data": [
    {
      "id": "c0a1b2c3-d4e5-4f60-8a71-92b3c4d5e6f7",
      "request_id": "0b1c2d3e-4f50-4617-8a29-3b4c5d6e7f80",
      "created_at": "2026-09-22T10:02:14Z",
      "token_group_id": "5f1c9d2e-7b3a-4c8e-9f21-0a6d4e8b1c33",
      "token_user_id": null,
      "token_key_id": "9b7e4a10-3c2d-4f5e-8a6b-1d2c3e4f5a60",
      "end_user_id": null,
      "requested_model": "Kimi-K3",
      "model": "Kimi-K3",
      "attempts": 1,
      "metadata": {},
      "input_tokens": 9,
      "output_tokens": 12,
      "cost": 0.000225,
      "status": "succeeded",
      "cache_hit": false,
      "usage_status": "known",
      "biller": "telnyx",
      "configuration_version": 1,
      "rate_version": "telnyx-reference-v1",
      "provider_key_id": null,
      "reservation_micro_usd": 0
    }
  ],
  "meta": {
    "page_number": 1,
    "page_size": 100,
    "has_more": false,
    "snapshot": "..."
  }
}
```

When `meta.has_more` is true, request the next `page[number]` with the same filters and `page[snapshot]=<meta.snapshot>`. See [Usage reporting](/docs/inference/ai-gateway/usage) for summaries grouped by group, user, key or end user.

`DELETE` requires the resource's current `ETag` in `If-Match`. Read the key first; the metadata GET never reveals the token.

```bash
KEY_ETAG=$(curl -sS -I "$AI_GATEWAY_MANAGEMENT_BASE_URL/token_keys/$TOKEN_KEY_ID" \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  | awk 'tolower($1) == "etag:" { sub(/\r$/, "", $2); print $2 }')

curl -X DELETE "$AI_GATEWAY_MANAGEMENT_BASE_URL/token_keys/$TOKEN_KEY_ID" \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "If-Match: $KEY_ETAG"
```

A `204` means the key is revoked: new requests with that token are rejected, while requests already admitted may finish. Spend history is retained. On `412`, the resource changed since the read; fetch it again rather than forcing the write.

The group stays available for new keys. To retire it as well, `DELETE /token_groups/{id}` with a fresh ETag and idempotency key. Deleting a group cascades to its keys.

## Next: use your own provider key

To run OpenAI or Anthropic models on your own provider account, store the provider key once, attach it to a group and allow a [bring-your-own-key model](/docs/inference/ai-gateway/inference-api#bring-your-own-key-models). Applications keep using the same token key. See [Bring your own key](/docs/inference/ai-gateway/byok).

## Next steps

    Available models, streaming, request limits and the supported request fields.
    Users, end users, PATCH semantics and pagination.
    What each limit does and how it is enforced.
    Status codes and structured error codes on both planes.
    Attach your own OpenAI or Anthropic key to a group.

---

## API

### Inference API

> Source: https://developers.telnyx.com/docs/inference/ai-gateway/inference-api.md

The inference plane serves three endpoints. All of them authenticate with an AI Gateway token key (`ltg_sk_...`) issued through the [management API](/docs/inference/ai-gateway/management-api). Telnyx account API keys, provider secrets and any other credential are rejected on this plane.

| Endpoint | Purpose |
| --- | --- |
| `GET https://llm.telnyx.com/v1/models` | Models available to the supplied token key. |
| `POST https://llm.telnyx.com/v1/chat/completions` | OpenAI-compatible Chat Completions, including streaming. Works with every model. |
| `POST https://llm.telnyx.com/v1/messages` | Anthropic-compatible Messages, including streaming. Only for [Anthropic BYOK models](#anthropic-sdk). |

Compatibility is bounded by the AI Gateway contract, not by every option the OpenAI and Anthropic SDKs expose; see [Supported request fields](#supported-request-fields) and [Supported Messages fields](#supported-messages-fields).

## Models

These Telnyx-hosted models are billed to your Telnyx account. Use these model names in the `model` field:

- `Kimi-K3`
- `Kimi-K2.6`
- `Kimi-K2.5`
- `GLM-5.3`
- `GLM-5.3-Flash`
- `GLM-5.2`
- `GLM-5.1-FP8`
- `MiniMax-M3-MXFP8`
- `MiniMax-M2.7`
- `Qwen3.8-27B`
- `Qwen3-235B-A22B`
- `DeepSeek-V4.1-Flash`
- `DeepSeek-V4-Flash-0731`
- `Llama-3.3-70B-Instruct`
- `Meta-Llama-3.1-70B-Instruct`
- `Meta-Llama-3.1-8B-Instruct`
- `gemma-2b-it`

A group lists the model names its keys may call in `allowed_models`; a key can narrow that list further. Per-request output and prompt limits are listed under [Request size limits](#request-size-limits).

### Bring-your-own-key models

These models run on your own provider account with a [provider key](/docs/inference/ai-gateway/byok) and are billed by that provider.

| Provider | Models |
| --- | --- |
| Anthropic | `claude-fable-5-1`, `claude-opus-5-5`, `claude-sonnet-5` |
| OpenAI | `gpt-6-astra`, `gpt-6-sol`, `gpt-6-luna`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano`, `gpt-4.1`, `gpt-4.1-mini`, `gpt-4o`, `gpt-4o-mini`, `o3` |

Add them to `allowed_models` like any other model. A BYOK model works only when the calling key's group has an attached provider key for that model's provider. Without one, requests fail with `503` and code `enforcement_unavailable`; they never fall back to a Telnyx-hosted model.

Every BYOK model works on `/v1/chat/completions`. Anthropic BYOK models also work on `/v1/messages`.

### Model discovery

`GET /v1/models` returns the models the supplied key may use: the intersection of the key's `allowed_models` and its group's list. It is not a global catalog, and it does not charge against any budget.

Hosted model entries include `capabilities`: `chat_tools`, `chat_tools_stream`, `chat_tool_choice`, `parallel_tool_calls`, `context_window`, `context_includes_output`, `context_enforcement`, `output_token_limit`, `default_output_tokens` and `max_request_body_bytes`. Context windows include input **and** output. `output_token_limit` is the model's explicit completion ceiling, or its total context bound when no separate ceiling exists. `default_output_tokens: null` means an omitted limit is left to the serving engine. `reservation_accounting` describes conservative budget/TPM accounting; it is not an exact tokenizer count. BYOK discovery entries retain their existing shape and may omit capabilities.

A newly created key can return `503` while its policy is being provisioned. Check the optional `x-ltg-enforcement-reason: provisioning` header and follow the [pending-provisioning guidance](/docs/inference/ai-gateway/errors#pending-provisioning) before sending inference requests.

```bash
curl https://llm.telnyx.com/v1/models \
  -H "Authorization: Bearer $AI_GATEWAY_TOKEN_KEY"
```

```json
{
  "object": "list",
  "data": [
    { "id": "Kimi-K3", "object": "model", "created": 1758535200, "owned_by": "telnyx" }
  ]
}
```

## OpenAI SDK

Point the OpenAI SDK at the inference base URL and use the token key as the API key.

```python Python
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
    base_url="https://llm.telnyx.com/v1",
    max_retries=0,
)

completion = client.chat.completions.create(
    model=os.environ["AI_GATEWAY_MODEL"],
    messages=[{"role": "user", "content": "Tell me about Telnyx"}],
    max_tokens=256,
    user="application-bound-user-id",
)
print(completion.choices[0].message.content)
```

```javascript JavaScript
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_TOKEN_KEY,
  baseURL: "https://llm.telnyx.com/v1",
  maxRetries: 0,
});

const completion = await client.chat.completions.create({
  model: process.env.AI_GATEWAY_MODEL,
  messages: [{ role: "user", content: "Tell me about Telnyx" }],
  max_tokens: 256,
  user: "application-bound-user-id",
});
console.log(completion.choices[0].message.content);
```

```bash curl
curl -X POST https://llm.telnyx.com/v1/chat/completions \
  -H "Authorization: Bearer $AI_GATEWAY_TOKEN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kimi-K3",
    "messages": [{"role": "user", "content": "Tell me about Telnyx"}],
    "max_tokens": 256,
    "user": "application-bound-user-id"
  }'
```

The optional `user` field attributes the request to an application end user for budgeting and reporting. See [End-user identity](/docs/inference/ai-gateway/controls#end-user-identity).

### Streaming

Set `stream: true` and consume the stream to completion. A stream that starts with HTTP 200 can still fail later; handle SDK exceptions and close the stream.

```python Python
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
    base_url="https://llm.telnyx.com/v1",
    max_retries=0,
)

with client.chat.completions.create(
    model=os.environ["AI_GATEWAY_MODEL"],
    messages=[{"role": "user", "content": "Tell me about Telnyx"}],
    max_tokens=256,
    stream=True,
) as stream:
    for chunk in stream:
        if chunk.choices:
            print(chunk.choices[0].delta.content or "", end="", flush=True)
```

```javascript JavaScript
const stream = await client.chat.completions.create({
  model: process.env.AI_GATEWAY_MODEL,
  messages: [{ role: "user", content: "Tell me about Telnyx" }],
  max_tokens: 256,
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
```

## Anthropic SDK

  `POST /v1/messages` is available **only for Anthropic models used with your own Anthropic key** ([Anthropic BYOK models](#bring-your-own-key-models)). A request to `/v1/messages` with a Telnyx-hosted model or an OpenAI BYOK model returns `400` with code `invalid_request`; call those models on `/v1/chat/completions`. Anthropic BYOK models work on both endpoints.

The Anthropic SDK appends `/v1/messages` to its base URL, so set the base URL to the gateway root **without** the `/v1` suffix. The SDK sends the token key in `x-api-key`, which the gateway accepts. `Authorization: Bearer` is also accepted; if both headers are present they must carry the same token.

```python Python
import os
from anthropic import Anthropic

client = Anthropic(
    api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
    base_url="https://llm.telnyx.com",
    max_retries=0,
)

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=256,
    messages=[{"role": "user", "content": "Tell me about Telnyx"}],
    metadata={"user_id": "application-bound-user-id"},
)
print(message.content[0].text)
```

```javascript JavaScript
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  apiKey: process.env.AI_GATEWAY_TOKEN_KEY,
  baseURL: "https://llm.telnyx.com",
  maxRetries: 0,
});

const message = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 256,
  messages: [{ role: "user", content: "Tell me about Telnyx" }],
  metadata: { user_id: "application-bound-user-id" },
});
console.log(message.content[0].text);
```

```bash curl
curl -X POST https://llm.telnyx.com/v1/messages \
  -H "x-api-key: $AI_GATEWAY_TOKEN_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "max_tokens": 256,
    "messages": [{"role": "user", "content": "Tell me about Telnyx"}],
    "metadata": {"user_id": "application-bound-user-id"}
  }'
```

Raw HTTP requests must send `anthropic-version: 2023-06-01`; the SDKs add it automatically. `metadata.user_id` and the OpenAI `user` field identify the same end-user namespace.

The endpoint streams standard Anthropic server-sent events (`message_start`, `content_block_start`, `content_block_delta`, `content_block_stop`, `message_delta`, `message_stop`). An error after the stream has started arrives as an `event: error` frame.

```python Python
with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=256,
    messages=[{"role": "user", "content": "Tell me about Telnyx"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
```

```javascript JavaScript
const stream = client.messages.stream({
  model: "claude-sonnet-5",
  max_tokens: 256,
  messages: [{ role: "user", content: "Tell me about Telnyx" }],
});

for await (const event of stream) {
  if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
    process.stdout.write(event.delta.text);
  }
}
```

### Supported Messages fields

| Field | Notes |
| --- | --- |
| `model` | Required. An Anthropic BYOK model available to this key. |
| `messages` | Required. 1 to 256 messages with roles `user` and `assistant`. |
| `max_tokens` | Required. At most 16,384. |
| `system` | String or array of `text` blocks. |
| `stream` | Anthropic server-sent events. |
| `temperature` | 0 to 1. |
| `top_p` | 0 to 1. |
| `top_k` | 0 to 1000. |
| `stop_sequences` | Up to 4. |
| `metadata` | Only `user_id` is accepted; it maps to the end-user dimension. |
| `tools`, `tool_choice` | Tool definitions with `name`, `description` and `input_schema`. `tool_choice` accepts `auto`, `any`, `tool`, `none` or an object. |

Content blocks of type `text`, `tool_use` and `tool_result` are supported. `image` blocks are rejected.

OpenAI-specific fields (`frequency_penalty`, `presence_penalty`, `n`, `seed`, `response_format`, `user`, `stream_options`, `parallel_tool_calls`) are rejected on the Messages endpoint.

Hosted Chat Completions tool calls are supported by the listed Kimi, GLM, MiniMax, Qwen and DeepSeek models, plus `Llama-3.3-70B-Instruct`. The two Meta Llama 3.1 models and `gemma-2b-it` reject tools before dispatch. Return the assistant's `tool_calls` message and matching `role: "tool"` messages with `tool_call_id` to continue the conversation. Preserve assistant `reasoning_content` when returned; raw responses and OpenAI SDK message serialization are accepted for these hosted models. The application executes the function; the gateway only carries its definition, call and result.

## Retries and timeouts

Inference is **not idempotent**. An SDK retry after a timeout or disconnect can dispatch a second billed request, and a timeout is not evidence that the first request did no provider work. The examples on this page set `max_retries=0`; if your application retries, do so deliberately and only for errors that occurred before dispatch (for example `401`, `403` or `429` with `Retry-After`). See [Errors](/docs/inference/ai-gateway/errors).

Token groups can configure [gateway retries and fallbacks](/docs/inference/ai-gateway/controls#fallback-models-and-retries). These request headers can tighten the policy without changing the group:

| Header | Accepted values |
| --- | --- |
| `x-ltg-max-attempts` | 1 to the group's `retries.max_attempts` (1 when disabled). |
| `x-ltg-retry-delay` | At least the configured base delay (250 ms when unset), at most 5,000 ms. |
| `x-ltg-backoff` | Equal or slower curve: `constant` < `linear` < `exponential`; default `exponential`. |
| `x-ltg-request-timeout` | 1,000–60,000 ms, no greater than the group's timeout when set or the attempt limit. Validated for all requests, applied only to streaming attempts until their first provider event. |

The other supported `x-ltg-*` request header is `x-ltg-metadata` for [request tags](/docs/inference/ai-gateway/usage#request-tags). Client-supplied fallback headers and other routing controls remain rejected.

The production request lifetime is 300 seconds, with each provider attempt bounded to 295 seconds. Ordinary streaming requests wait up to 30 seconds for the first provider event; large-prompt requests can wait up to the attempt limit. Subsequent stream reads have a 30-second idle timeout. Group and request policies can shorten the first-event wait. Client-side timeouts do not extend these limits.

## Request size limits

Hosted models use their published total context windows. Input, tool definitions, conversation history and requested output must fit the model's context. The actual serving tokenizer enforces this limit; discovery reports the active published values and request-body envelope.

| Models | Total context tokens |
| --- | ---: |
| `Kimi-K3`, `GLM-5.2`, `MiniMax-M3-MXFP8` | 1,000,000 |
| `GLM-5.3`, `GLM-5.3-Flash`, `DeepSeek-V4.1-Flash`, `DeepSeek-V4-Flash-0731` | 1,048,576 |
| `Kimi-K2.6`, `Qwen3.8-27B` | 262,144 |
| `Kimi-K2.5` | 256,000 |
| `GLM-5.1-FP8` | 202,752 |
| `MiniMax-M2.7` | 200,000 |
| `Meta-Llama-3.1-8B-Instruct` | 131,072 |
| `Llama-3.3-70B-Instruct`, `Meta-Llama-3.1-70B-Instruct` | 99,000 |
| `Qwen3-235B-A22B` | 32,768 |
| `gemma-2b-it` | 8,192 |

Hosted output limits follow the model-specific [Telnyx Chat Completions policy](/api-reference/openai-chat/create-a-chat-completion-openai-compatible). The output budget includes reasoning and answer tokens combined.

| Models | Explicit completion ceiling |
| --- | ---: |
| `Kimi-K3`, `DeepSeek-V4.1-Flash` | 64,000 |
| `GLM-5.2` | 131,072 |
| Other hosted models | No separate completion ceiling; input and output together must fit the total context. |

An explicit limit above a declared completion ceiling is clamped to that ceiling. Without a separate ceiling, an output request larger than the total context is rejected. The total context is not an available output allowance after a prompt has consumed part of it.

An omitted or null limit defaults to 8,192 for `GLM-5.3`, `GLM-5.3-Flash` and `DeepSeek-V4.1-Flash`. Other hosted models leave the limit to the serving engine. Use `GET /v1/models` to inspect the active limits and defaults for the calling key.

BYOK retains its existing conservative prompt allowance (about 128k, or 96k for `gpt-4o`/`gpt-4o-mini`) and 16,384 output-token cap; the larger hosted windows do not expand BYOK limits.

A known backend context rejection returns `400 context_length_exceeded`: reduce input and/or requested output. It is terminal and does not trigger retry or fallback. The separate physical JSON-body limit reported by `max_request_body_bytes` includes escaping, tools and history; exceeding it returns `413 request_too_large`. Published context is not a guarantee of capacity or successful completion at that size on every serving backend. Timeouts return `504 upstream_timeout`, and other upstream failures can return `502 upstream_error`.

Set `max_tokens` (or `max_completion_tokens`) explicitly to bound generation and reservations. For hosted models that leave an omitted or null limit to the engine, the gateway reserves the **full context window as output**, even if an explicit request would have a separate completion ceiling. For example, Kimi-K3 reserves 1,000,000 output tokens when the limit is omitted. Models with an 8,192 default reserve that default; explicit limits reserve the resolved limit after clamping.

Budget and TPM accounting also includes a conservative input bound from normalized JSON bytes plus 256 per message. These reservations are not actual token counts or charges. Omitted limits can exhaust budget/TPM headroom or reduce the number of concurrent requests that fit within a budget. Configured retries/fallbacks can add further reservations. See [Reservations](/docs/inference/ai-gateway/controls#reservations).

Output-token admission does not guarantee delivery of that many tokens. Hosted buffered Chat responses can be up to 128 MiB, subject to available buffering capacity; responses above 1 MiB are not cached. BYOK buffered responses remain limited to 1 MiB. Hosted streams have a 1 MiB per-event limit without an aggregate response-size cap; BYOK retains its 8 MiB aggregate limit. Enabled response guardrails limit buffered evaluation to 1 MiB and stream retention to 8 MiB, including passthrough streaming; exceeding these limits fails the request. Server-side request lifetimes still apply; see [Retries and timeouts](#retries-and-timeouts).

## Supported request fields

Unknown top-level fields and options rejected by gateway validation return `400`. Model-specific validation may also happen upstream; upstream failures can return `502 upstream_error`.

### Chat Completions

| Field | Notes |
| --- | --- |
| `model` | Required. Must be a model returned by `GET /v1/models` for this key. |
| `messages` | Required. Roles: `system`, `developer`, `user`, `assistant`, `tool`. Content is a string or an array of `text` and `image_url` parts. |
| `max_tokens` / `max_completion_tokens` | Send one or the other, never both. Hosted ceilings and omitted/null defaults are model-specific. BYOK remains capped at 16,384. Explicit limits are strongly recommended; see [Request size limits](#request-size-limits). |
| `stream`, `stream_options` | Server-sent events. |
| `temperature` | 0 to 2. |
| `top_p` | 0 to 1. |
| `stop` | String or up to 4 strings. |
| `seed`, `presence_penalty`, `frequency_penalty` | Passed through where the model supports them. |
| `n` | Only `1` is supported. |
| `user` | End-user identifier for budgeting and reporting. |
| `tools`, `tool_choice`, `parallel_tool_calls` | Function tools where advertised by model discovery. Hosted tool-capable models accept `auto` or `none`, streamed calls and tool-result continuation; `parallel_tool_calls` must be false (also the effective default with tools). `required`, named selection and parallel calls are rejected for this hosted profile. BYOK capabilities remain model-dependent. |
| `response_format` | Set `type` to `json_object` for a JSON object, or `json_schema` with `json_schema.name`, `json_schema.schema` and optional `json_schema.strict` for a supplied schema. Support depends on the model and `stream`; see below. |

  **Structured JSON on Telnyx-hosted models:** All 17 models support `json_schema` without streaming. All except `Meta-Llama-3.1-8B-Instruct` support `json_object`. Both formats support `stream: true` where enabled, except on `Llama-3.3-70B-Instruct`. Unsupported combinations return `400` before inference.

  `json_object` does not enforce requested fields or values. Use `json_schema` when a specific structure is required. Schemas are limited to 16 KiB, 16 levels and 512 schema nodes; remote references are rejected. Handle truncation, refusals and stream errors, and validate the completed output.

Image parts must be inline `data:image/...;base64,...` URLs (PNG, JPEG, GIF or WebP). Remote image URLs are rejected.

### Not supported

- Custom provider URLs, provider credentials, routing or fallback controls in the request body.
- Remote image URLs.
- Unlisted beta headers and options.
- The Responses API, embeddings, image and audio generation, batches and realtime APIs. These are not part of the AI Gateway surface; see the [Telnyx Inference API](/docs/inference/getting-started) for those capabilities.

---

### Management API

> Source: https://developers.telnyx.com/docs/inference/ai-gateway/management-api.md

The management plane lives at `https://api.telnyx.com/v2/llm_token_gateway` and authenticates with a Telnyx account API key:

```text
Authorization: Bearer $TELNYX_API_KEY
```

All paths on this page are relative to that base. Every resource is scoped to the authenticated account; a resource that belongs to another account returns `404`.

## Conventions

### Response envelope

Success bodies wrap the resource in `data`. List responses add `meta` with pagination state. Every response carries an `X-Request-ID` correlation header; keep it when reporting a problem. Responses are `Cache-Control: no-store`.

### Idempotency

Every `POST`, `PATCH`, `PUT` and `DELETE` requires an `Idempotency-Key` header. Keys are scoped to account, method and path and retained for 24 hours.

- The same key with the same body returns the original outcome.
- The same key with a different body returns `409` with code `idempotency_conflict`.
- A key whose original request is still in progress returns `409` with `Retry-After`.
- A replayed token-key create returns `200` with metadata only; the secret is never redisclosed. A provider-key secret is never returned, including on replay.

When a mutation times out, retry it with the **same** key and body. Generating a fresh key starts a new operation and can create a duplicate resource.

### ETag preconditions

Every resource carries an integer `version`, returned as a quoted `ETag` header on `GET`, create and update responses. `PATCH` and `DELETE` require that value in `If-Match`:

| Condition | Response |
| --- | --- |
| `If-Match` matches the current version | The mutation is applied. |
| `If-Match` is stale | `412` with code `precondition_failed`. Read the resource again. |
| `If-Match` is missing | `428` with code `precondition_required`. |

End-user caps use `PUT` as a full replacement: send `If-None-Match: *` to create and the current `If-Match` to replace.

### PATCH semantics

- A field omitted from a `PATCH` body is preserved.
- A nullable field set to `null` is cleared (for example `max_budget: null` removes a group or user cap). On a token key, a null limit is reset to its maximum instead; see [Token keys](#token-keys).
- Policy changes apply to new admissions. They do not erase spend history or unresolved exposure, and a request already admitted under the old policy may finish.
- A change can take a short time to propagate. During that window the management response is already committed, but inference may return `503` with code `enforcement_unavailable`. Do not treat a pending change as permission to rely on the old policy.

### Pagination

Resource list endpoints accept `page[number]`, `page[size]` (1 to 100) and `page[snapshot]`. [Guardrail events](/docs/inference/ai-gateway/guardrails#query-findings) use offset pagination without a snapshot.

```json
{
  "data": [ ... ],
  "meta": {
    "page_number": 1,
    "page_size": 100,
    "has_more": true,
    "snapshot": "..."
  }
}
```

The first page returns a `meta.snapshot` bound to the account and filters and valid for 15 minutes. To fetch later pages, increment `page[number]` and pass `page[snapshot]=<meta.snapshot>` with the **same filters**. A missing, expired or mismatched snapshot returns `409`.

## Token groups

A group defines the model allowlist and shared limits for the keys inside it.

| Operation | Path |
| --- | --- |
| Create | `POST /token_groups` |
| List | `GET /token_groups` |
| Read | `GET /token_groups/{id}` |
| Update | `PATCH /token_groups/{id}` |
| Delete | `DELETE /token_groups/{id}` |

| Field | Type | Notes |
| --- | --- | --- |
| `name` | string | Required. 1 to 256 characters. |
| `allowed_models` | string[] | Required. Up to 1000 unique [model names](/docs/inference/ai-gateway/inference-api#models). An empty list permits no inference. |
| `max_budget` | number or null | USD, at most six decimal places. Null is uncapped; zero denies paid requests. |
| `budget_duration` | `1d`, `7d`, `30d` or null | Anchored budget period. Null means a lifetime budget. |
| `rpm_limit`, `tpm_limit` | integer or null | Requests and tokens per rolling 60-second window. Null removes the limit; zero denies. |
| `max_parallel_requests` | integer or null | 1–1,000 concurrent cache misses across the group. Null is uncapped. See [Concurrent requests](/docs/inference/ai-gateway/controls#concurrent-requests). |
| `cache_ttl` | integer or null | 1–86,400 seconds; null disables [response caching](/docs/inference/ai-gateway/controls#response-caching). |
| `fallbacks`, `retries` | object or null | Ordered model fallbacks and bounded same-model retry policy. See [Fallback models and retries](/docs/inference/ai-gateway/controls#fallback-models-and-retries). |
| `provider_key_ids` | string[] | [Provider keys](#provider-keys) for [BYOK models](/docs/inference/ai-gateway/inference-api#bring-your-own-key-models). Defaults to `[]`. At most one key per provider; each key must belong to the same account (otherwise `404`) and its provider must match at least one BYOK model in `allowed_models` (otherwise `400`). Two keys for the same provider return `409`. |
| `blocked` | boolean | Blocks every key in the group. |
| `guardrails` | object or null | Group policy for secret and sensitive-data inspection. Defaults to null (disabled). See [Guardrails](/docs/inference/ai-gateway/guardrails) for actions, DLP profiles and streaming behavior. |

Read-only fields on the response: `id`, `version`, `spend`, `reserved_spend`, `budget_started_at`, `resets_at`, `created_at`, `updated_at`.

Deleting a group revokes its keys and removes user memberships; spend history is retained.

## Token users

A user represents an application actor that may belong to more than one group. Limits set on the user aggregate across all of its keys in every group.

| Operation | Path |
| --- | --- |
| Create | `POST /token_users` |
| List | `GET /token_users` |
| Read | `GET /token_users/{id}` |
| Update | `PATCH /token_users/{id}` |
| Delete | `DELETE /token_users/{id}` |

| Field | Type | Notes |
| --- | --- | --- |
| `name` | string | Required. |
| `token_group_ids` | string[] | Required. Groups this user may hold keys in. |
| `external_id` | string or null | Your own identifier for the actor. |
| `max_budget`, `budget_duration` | | Aggregate budget across the user's keys. |
| `rpm_limit`, `tpm_limit` | | Aggregate rate limits across the user's keys. |

Removing a group from `token_group_ids` while the user still holds active keys in that group returns `409`; revoke those keys first. Deleting a user revokes its keys and retains spend history.

## Token keys

A key is the credential an application presents to the inference plane.

| Operation | Path |
| --- | --- |
| Create | `POST /token_keys` |
| List | `GET /token_keys` (filters: `token_group_id`, `token_user_id`) |
| Read | `GET /token_keys/{id}` |
| Update | `PATCH /token_keys/{id}` |
| Revoke | `DELETE /token_keys/{id}` |

| Field | Type | Notes |
| --- | --- | --- |
| `name` | string | Required. |
| `token_group_id` | uuid | Required. Cannot be changed after creation. |
| `token_user_id` | uuid or null | Null creates a service key. The user must be a member of the group. Cannot be changed after creation. |
| `allowed_models` | string[] or null | Null inherits the group's list. An empty list denies every model. A non-empty list narrows the group's list. |
| `max_budget` | number or null | USD budget scoped to this key, above 0 and at most 1000, with at most six decimal places. Omitted or null is stored as 1000. |
| `budget_duration` | `1d`, `7d`, `30d` or null | Null means a lifetime budget. |
| `rpm_limit` | integer or null | 1 to 6000. Omitted or null is stored as 6000. |
| `tpm_limit` | integer or null | 1 to 10000000. Omitted or null is stored as 10000000. |
| `max_parallel_requests` | integer or null | 1–1,000 concurrent cache misses for this key. Null means no key cap; the group cap still applies. PATCH null clears this cap. |
| `expires_at` | date-time or null | After this instant the key is rejected. |
| `blocked` | boolean | Rejects new requests without deleting the key. |
| `required_end_user_id` | boolean | Requires a non-empty `user` / `metadata.user_id` on every request. Presence only, not authenticity. |

Key budget and rate limits differ from group and user limits: a key is never uncapped for those controls and cannot be denied with a zero limit. A value of 0 or above the maximum returns `400` with code `limit_out_of_range`; a negative, fractional or non-numeric value, or a budget with more than six decimal places, returns `400` with code `invalid_request`. On a blocked key, a null limit is kept until the key is unblocked. To deny a key, set `blocked: true` or revoke it. See [Token key limits](/docs/inference/ai-gateway/controls#token-key-limits).

The create response is the **only** place `data.token` appears. It matches `^ltg_sk_[A-Za-z0-9_-]+$`. Store it immediately; a lost token cannot be recovered, only replaced.

Token-key create, read and update responses can include advisory `provisioning`: `ready`, `pending`, `unavailable`, `inactive`, `unsupported` or `unknown`. `ready` reflects the last readiness check; it is not a guarantee that inference will succeed. `pending` or `unknown` means activation is incomplete or unconfirmed. Use bounded retries of model discovery before starting inference; see [Pending provisioning](/docs/inference/ai-gateway/errors#pending-provisioning). Policy changes can temporarily make an existing key unavailable while its new configuration is applied.

`DELETE` revokes the key. Acknowledged revocation blocks new admissions; requests already admitted may complete.

## End users

An end-user cap applies an account-scoped budget or block to a caller-asserted identifier: the value an application sends as OpenAI `user` or Anthropic `metadata.user_id`. The identifier is the resource ID.

| Operation | Path |
| --- | --- |
| List | `GET /end_users` |
| Read | `GET /end_users/{id}` |
| Create or replace | `PUT /end_users/{id}` |
| Delete | `DELETE /end_users/{id}` |

| Field | Type | Notes |
| --- | --- | --- |
| `max_budget` | number or null | Required in the body. Null is uncapped. |
| `budget_duration` | `1d`, `7d`, `30d` or null | Required in the body. |
| `blocked` | boolean | Required in the body. Denies every request carrying this identifier. |

`PUT` is a full replacement and returns `200` for both create and replace. Send `If-None-Match: *` to create and the current `If-Match` to replace; a missing precondition returns `428`.

End-user identifiers are assertions made by whoever holds the token key. Bind them to authenticated users in a trusted backend; see [End-user identity](/docs/inference/ai-gateway/controls#end-user-identity).

## Provider keys

A provider key stores your own OpenAI or Anthropic secret for [bring your own key](/docs/inference/ai-gateway/byok). Attach it to groups through `provider_key_ids`.

| Operation | Path |
| --- | --- |
| Create | `POST /provider_keys` |
| List | `GET /provider_keys` |
| Read | `GET /provider_keys/{id}` |
| Delete | `DELETE /provider_keys/{id}` |

| Field | Type | Notes |
| --- | --- | --- |
| `name` | string | Required. 1 to 256 characters. |
| `provider` | `openai` or `anthropic` | Required. Any other value returns `400`. URLs are never accepted. |
| `secret` | string | Required on create, write-only. Never returned by any response, list or replay. |

Read-only fields on the response: `id`, `version`, `created_at`, `updated_at`.

Provider keys cannot be edited; `PATCH` returns `405`. To change a secret, create a new provider key, attach it to the groups, then delete the old one.

`DELETE` requires the current ETag in `If-Match`, like other deletes. Deleting a provider key detaches it from every group that references it; requests already in progress complete.

## Usage

`GET /usage/summary` returns group totals, daily and model breakdowns, complete guardrail counts and recent findings in one response. It requires a group and date range and has no pagination. `GET /spend/events` and `GET /spend/summary` provide request-level and dimensional reporting with snapshot pagination. See [Usage reporting](/docs/inference/ai-gateway/usage).

## Guardrail findings

`GET /guardrail_events` lists account-scoped flagged and blocked findings. Filter by date range, group, key, end user, stage or outcome. This endpoint uses offset pagination without `page[snapshot]`; see [Query findings](/docs/inference/ai-gateway/guardrails#query-findings).

---

## Concepts

### Controls

> Source: https://developers.telnyx.com/docs/inference/ai-gateway/controls.md

Every inference request passes an admission check before it is dispatched to a model. Admission evaluates the token key, its user, its group and the asserted end user together; the request is denied if any scope fails. This page describes each control and its observable behavior.

## Enforcement scopes

| Scope | Set on | Applies to |
| --- | --- | --- |
| Key | `POST /token_keys` | Requests made with that key. |
| User | `POST /token_users` | All keys owned by that user, across every group it belongs to. |
| Group | `POST /token_groups` | All keys in the group. |
| End user | `PUT /end_users/{id}` | Every request in the account that asserts that end-user identifier. |

Model access is the intersection of the group's `allowed_models` and the key's `allowed_models` (when set). A blocked key, user, group or end user denies the request with `403`.

## Budgets

Budgets are USD amounts with at most six decimal places, enforced independently at the key, user, group and end-user scopes.

- On groups, users and end users, `max_budget: null` is uncapped and `max_budget: 0` denies every paid request. Token keys differ; see [Token key limits](#token-key-limits).
- `budget_duration` of `1d`, `7d` or `30d` starts an anchored period at the moment the budget is committed. Periods roll from that anchor, not from midnight or the calendar month. A null duration is a lifetime budget.
- Changing the amount keeps the current period. Changing the duration starts a new anchored period. Neither change erases history.
- The response fields `spend`, `reserved_spend`, `budget_started_at` and `resets_at` on each resource show the current period.

### Reservations

Before a request is dispatched, the gateway reserves a conservative upper bound on its cost from the input size and the resolved output allowance. Explicit `max_tokens` or `max_completion_tokens` bounds that allowance. Hosted models that leave an omitted or null limit to the engine reserve their full context window as output; models with an 8,192 default reserve that default. See [Output limits and defaults](/docs/inference/ai-gateway/inference-api#request-size-limits). If any scope lacks that much headroom, the request is denied with `403` and code `budget_exceeded` (or `end_user_budget_exceeded`). The reservation is not shrunk to fit.

Multiple inference requests can use the same token key concurrently. Concurrent cache misses each reserve their own cost bound and must pass the applicable budget and rate limits. A request with unknown usage retains its monetary reservation without blocking later requests that fit within the remaining limits.

After the response completes, the reservation is replaced by the actual cost. If the outcome is unknown, for example because the stream was interrupted before usage was reported, the reservation is retained as exposure and the spend event reports `cost: null`. Unknown usage is not zero usage, and a timeout is not a refund.

Budgets are an application control, not an absolute guarantee of spend.

### Budgets and billing

- Usage of Telnyx-hosted models is billed to your Telnyx account at standard [Telnyx AI Inference pricing](https://telnyx.com/pricing/inference-api) for each model.
- Requests on [bring-your-own-key models](/docs/inference/ai-gateway/byok) are billed by your provider on your provider account, not by Telnyx. Budgets and rate limits still apply and act as a guard on that provider spend.
- Budgets and the `cost` values in usage reporting are measured at a flat reference rate of USD 5 per million input tokens and USD 15 per million output tokens, for enforcing limits and attribution. They are not your invoice.

## Rate limits

`rpm_limit` and `tpm_limit` are evaluated over rolling 60-second windows, not calendar minutes, at the key, user and group scopes.

- On groups and users, null removes the limit and zero denies every request. Token keys differ; see [Token key limits](#token-key-limits).
- Temporary rate contention returns `429` with a `Retry-After` header.
- A single request whose reservation exceeds an entire configured TPM limit returns `400 request_exceeds_tpm_limit` without `Retry-After`, before provider dispatch. Set an explicit output limit, reduce input or planned attempts, or ask the policy owner to adjust the limit. Retrying the unchanged request cannot succeed.
- `tpm_limit` counts reserved input and output tokens. An omitted or null hosted output limit can reserve the full context window as output. Set `max_tokens` or `max_completion_tokens` explicitly to bound the reservation; the input allowance must also fit within `tpm_limit`.

## Concurrent requests

Set `max_parallel_requests` on a token group or token key to an integer from 1 to 1,000. Null (the default) means no concurrency cap at that scope; omission in a PATCH preserves the current value and null clears it. Both limits apply. The group limit covers all its keys across gateway replicas. Token users and end users do not accept this field.

A slot covers an admitted cache miss until settlement or release; cache hits bypass the cap. Lowering a limit does not cancel requests already admitted. New requests above the limit receive `429 concurrency_limit_exceeded`, `errors[0].meta.scope` (`token_key` or `token_group`) and `Retry-After: 1`. No provider call, budget reservation or rate allocation is made for a denied request. The retry delay is advisory, not a promised completion time.

Unknown usage keeps its monetary hold without a concurrency slot. After a crashed owner, a slot can remain occupied until recovery or lease expiry (up to five minutes). This control bounds gateway-owned requests, not remote provider activity after a lost connection.

## Fallback models and retries

Configure `fallbacks` and `retries` on the token group. Both apply to Telnyx-hosted routes and default to disabled. BYOK requests do not use them.

```json
{
  "allowed_models": ["Kimi-K3", "GLM-5.3-Flash"],
  "fallbacks": {"Kimi-K3": ["GLM-5.3-Flash"]},
  "retries": {
    "max_attempts": 2,
    "retry_delay_ms": 250,
    "backoff": "exponential",
    "request_timeout_ms": 10000
  }
}
```

This is a partial group PATCH body; use the current ETag and an idempotency key as described under [Mutation safety](#mutation-safety).

| Field | Contract |
| --- | --- |
| `fallbacks` | Primary model name mapped to an ordered list of 1–3 other model names. Every name must appear in the group's `allowed_models`; the calling key must also permit the destination. No duplicates, self-reference or wildcards. A destination's own chain is not expanded. Null or `{}` disables fallbacks. |
| `retries.max_attempts` | Required when retries is an object; 1–5 attempts per model, including the first attempt. |
| `retries.retry_delay_ms` | 0–5,000 milliseconds; defaults to 250. |
| `retries.backoff` | `constant`, `linear` or `exponential`; defaults to `exponential`. |
| `retries.request_timeout_ms` | Null or 1,000–60,000 milliseconds to the first provider event of each **streaming** attempt. Never applies after the first event; does not shorten non-stream requests. |

`max_attempts × (1 + longest fallback chain)` must not exceed eight. Invalid configuration, including removing an allowed model still referenced by a chain, returns `400`. PATCH omission preserves either policy; `retries: null` disables retries.

The gateway retries the same model only when it definitely did no provider work: a connection failure before dispatch, or provider `429`/`529`. An uncertain failure such as a timeout or `408`/`500`/`502`/`503`/`504` moves to the next fallback instead. Retry waits follow the selected curve, are capped at five seconds and jittered between half and the full delay. A provider `Retry-After` of at most five seconds is honored; a longer delay moves to the next step. The request deadline still applies.

Fallback destinations are checked against their capabilities and request-size limits. A context-length rejection is terminal. Once the first provider event is forwarded, the model never changes and no retry or fallback occurs. If every attempt fails before that event, the response is HTTP `502 upstream_error`, or `504 upstream_timeout` when the last attempt timed out. An error after streaming begins is an error event followed by closure, without a success terminator.

The response `model` and `x-ltg-served-model` identify the serving model; `x-ltg-attempts` counts provider attempts (zero for a cache hit). Spend events include `requested_model`, the serving `model` and `attempts`. Tokens and spend include all attempts. Budget and TPM capacity for planned attempts are reserved up front; unused reservations are released at settlement. Unknown usage retains its monetary hold even if a later attempt succeeds.

SDK retries are additional requests with their own attempts and reservations. Set SDK retries to zero when relying on gateway retries. Request headers can tighten the group policy; see [Retries and timeouts](/docs/inference/ai-gateway/inference-api#retries-and-timeouts).

## Response caching

Set group `cache_ttl` to 1–86,400 seconds to enable response caching, or null to disable it. Omission in a PATCH preserves the current setting. Caching is best effort; entries can be evicted before their TTL, and equivalent requests on different replicas can miss.

Only complete supported responses are cached. Partial streams, failures and responses that cannot be reconstructed safely are excluded. Cache identity includes the applicable key, group, user, model, provider route and policy. A policy change prevents reuse under the old policy. Cache hits avoid provider work, report zero reference cost and provider tokens, and do not consume concurrency slots. Authorization and policy checks still apply.

Send `"cache": {"no-cache": true}` in the inference request to bypass caching. Request metadata tags are not part of cache identity; a hit is attributed to the tags on the current request. Caching does not configure a provider's prompt cache.

## Token key limits

Token keys always have budget and rate caps. The optional concurrency cap is separate. Group and user budget/rate limits are unchanged: null is uncapped and zero denies.

| Field | Range | Omitted or null |
| --- | --- | --- |
| `max_budget` | Above 0, at most USD 1,000, up to six decimal places | USD 1,000 |
| `rpm_limit` | 1 to 6,000 | 6,000 |
| `tpm_limit` | 1 to 10,000,000 | 10,000,000 |

- An omitted or null key limit is stored as its maximum. On a blocked key, a null limit is kept until the key is unblocked.
- The default USD 1,000 budget is a lifetime budget unless `budget_duration` is set.
- A value of 0 or above the maximum returns `400` with code `limit_out_of_range`.
- A negative, fractional or non-numeric value, or a budget with more than six decimal places, returns `400` with code `invalid_request`.
- To deny a key, block it (`blocked: true`) or revoke it.

## End-user identity

The OpenAI `user` field and the Anthropic Messages `metadata.user_id` field identify an account-scoped end user in the same namespace. The value is used for end-user budgets and blocks and appears as `end_user_id` in usage reporting.

The identifier is an assertion by whoever holds the token key, not an authenticated identity. A key embedded in a client can assert any value. Bind end-user identifiers to authenticated sessions in a trusted backend, and set `required_end_user_id: true` on a key when every request must carry one. That flag checks presence only.

## Mutation safety

Management mutations are protected by idempotency keys and ETag preconditions; see [Conventions](/docs/inference/ai-gateway/management-api#conventions). A committed policy change applies to new admissions once it has propagated. During propagation, inference can return `503` with code `enforcement_unavailable`. Fail closed in that case rather than retrying under the assumption that the previous policy still applies.

## Content guardrails

Set a group-level `guardrails` policy to flag or block supported secrets and sensitive data before provider dispatch or before delivering a response. Prompt blocks reserve no budget; response blocks still account for provider work. See [Guardrails](/docs/inference/ai-gateway/guardrails) for configuration and streaming behavior.

---

### Guardrails

> Source: https://developers.telnyx.com/docs/inference/ai-gateway/guardrails.md

Guardrails inspect supported text fields in prompts and model responses. Configure a policy on a [token group](/docs/inference/ai-gateway/management-api#token-groups); every token key in that group uses it. Policies apply to new admissions. Existing groups have `guardrails: null`, which disables inspection.

Guardrails detect supported credential and sensitive-data patterns. They do not redact text or provide model-based content moderation, prompt-injection detection, or a guarantee that all sensitive data will be detected.

## Configure a policy

Include `guardrails` when creating a token group, or send it in `PATCH /v2/llm_token_gateway/token_groups/{id}`. Authenticate with a Telnyx account API key and supply the [idempotency and ETag headers](/docs/inference/ai-gateway/management-api#etag-preconditions) required for the mutation.

This PATCH body blocks supported secrets in both directions and flags financial data:

```json
{
  "guardrails": {
    "secrets": {
      "prompt": "block",
      "response": "block"
    },
    "dlp": {
      "profiles": ["financial"],
      "prompt": "flag",
      "response": "flag"
    },
    "streaming": "buffered"
  }
}
```

Each detector has independent `prompt` and `response` actions:

| Action | Behavior |
| --- | --- |
| `ignore` | Do not run the detector at that stage. This is the default. |
| `flag` | Allow the request or response and record findings. |
| `block` | Reject a matching prompt or withhold a matching response. |

If several rules match, `block` takes precedence over `flag`. DLP needs at least one profile whenever either DLP action is `flag` or `block`. Unknown fields, duplicate profiles and unsupported values return `400`.

Omitting `guardrails` from a PATCH preserves the policy. Supplying a `guardrails` object replaces the entire nested policy: omitted actions become `ignore`, and omitted DLP profiles become an empty list. Send the complete intended policy when changing one action. To disable inspection, send:

```json
{
  "guardrails": null
}
```

A policy update can briefly cause inference to return `503` while the change propagates. Read the group back to verify the saved policy.

## Detectors

| Detector or profile | Supported patterns |
| --- | --- |
| `secrets` | AWS access key IDs; supported GitHub, Slack, Stripe, OpenAI, Anthropic, Google and Telnyx key patterns; AI Gateway token keys; JWT-shaped tokens; private-key block markers. |
| DLP `financial` | Luhn-valid card-number patterns and checksum-validated IBAN patterns. |
| DLP `government_id` | US Social Security number patterns and UK National Insurance number patterns. |
| DLP `contact` | Email address and E.164 phone-number patterns. |

These are pattern checks, not verification that a credential is active or an identifier belongs to a real person. Formats outside the supported patterns can be missed; text that resembles a supported format can match. Names, addresses and other free-text personal information are not detected.

The gateway inspects supported message text, system text, tool definitions and tool inputs/arguments. Response inspection includes supported text, refusal, thinking and tool-use fields. Opaque provider signatures, encrypted thinking data, provider metadata and unrecognized content-block types are outside this inspection coverage. Guardrails do not inspect images or audio.

## Blocking and usage

| Stage | Observable result | Usage behavior |
| --- | --- | --- |
| Prompt | HTTP `400`, code `prompt_blocked`. | No provider request or budget reservation; no spend event is created. |
| Complete response | HTTP `400`, code `response_blocked`. | The provider already generated the response. Its usage is accounted for even though the response is withheld. |
| Buffered streamed response | An SSE error carrying `response_blocked`, after the HTTP `200` headers. No provider content is released. | Provider usage is accounted for. |

Guardrail findings are separate from the [usage ledger](/docs/inference/ai-gateway/usage). Do not interpret a blocked response as a refund or automatically retry it. When a cached response is available, it is checked against the current response policy before replay; a blocked cache hit makes no provider call and creates no spend event.

## Streaming

`streaming` accepts `buffered` or `passthrough`:

- **Buffered:** the gateway holds the stream until inspection completes, then replays its SSE events. This delays the first content event. Any response `block` action requires this mode; it is selected automatically if `streaming` is omitted. Explicit `passthrough` with a response `block` action is rejected with `400`.
- **Passthrough:** events are forwarded immediately and findings are evaluated after the stream completes. This mode cannot block responses. It is the default when no response action is `block`.

For Chat Completions, a streamed block is a `data:` event containing the error document. For Anthropic Messages, it is an `event: error` frame. Handle stream errors even when the initial HTTP status was `200`.

## Flagged-response header

Successful responses with flagged findings can include `x-ltg-policy`. Its value is JSON containing detector codes, counts and actions:

```json
{
  "outcome": "flagged",
  "findings": [
    {
      "detector": "dlp",
      "code": "email",
      "count": 1,
      "action": "flag"
    }
  ]
}
```

Prompt findings are available on successful complete and streamed responses. Response findings are available in the header only for non-streamed responses; streamed response findings are recorded after the headers have been sent. Blocked findings are available through the events endpoint, not this header.

Finding counts represent pattern matches during inspection, not unique sensitive values. Structured data can be inspected in both its JSON spelling and unescaped form, so a value can contribute more than one match.

## Query findings

Use `GET https://api.telnyx.com/v2/llm_token_gateway/guardrail_events` with the Telnyx account API key. Only that account's events are returned, newest first. Events identify the request, token key, group, optional user/end user, model, stage, outcome and findings. Findings contain codes and counts, never matched text or prompt/response excerpts.

| Query parameter | Values or behavior |
| --- | --- |
| `start_date`, `end_date` | Supply both as `YYYY-MM-DD`. Start is inclusive and end exclusive, in UTC; the range must be positive and at most 31 days. With neither supplied, the range covers today and the previous six UTC dates. |
| `token_group_id`, `token_key_id` | Filter by UUID. An unknown or foreign-account ID returns no matching events. |
| `end_user_id` | Filter by the asserted end-user identifier. |
| `stage` | `prompt` or `response`. |
| `outcome` | `flagged`, `blocked`, `evaluated` or `unevaluated`. The current deterministic detectors emit `flagged` and `blocked`; clean passes create no event. |
| `page[number]` | Positive page number; defaults to `1`. |
| `page[size]` | `1` to `100`; defaults to `20`. |

The response contains `data` and `meta.page_number`, `meta.page_size`, `meta.has_more`. Unlike resource and spend listings, guardrail events use offset pagination and do not accept `page[snapshot]`. Newly inserted events can shift page boundaries; deduplicate by event `id` when collecting several pages.

A findings-storage failure does not make a blocked request pass, but it can leave no event for that decision. Use the inference error and `X-Request-ID` as well when investigating a blocked request.

---

### Bring your own key

> Source: https://developers.telnyx.com/docs/inference/ai-gateway/byok.md

By default, a token group uses Telnyx-hosted models and usage is billed to your Telnyx account. With bring your own key (BYOK), you store an OpenAI or Anthropic key through the management API and attach it to one or more groups. Requests on [bring-your-own-key models](/docs/inference/ai-gateway/inference-api#bring-your-own-key-models) from those groups run on your provider account, and your provider bills them.

BYOK changes **who is billed**. It does not change attribution, budget enforcement or rate limiting: key, user, group and end-user controls apply exactly as they do for Telnyx-hosted models.

## Set up BYOK

`POST /provider_keys` with a `name`, the `provider` (`openai` or `anthropic`) and the `secret`. The secret is accepted only on create and is never returned by any response, list or idempotent replay.

```bash
curl -X POST https://api.telnyx.com/v2/llm_token_gateway/provider_keys \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{
    "name": "anthropic-production",
    "provider": "anthropic",
    "secret": "<provider-secret>"
  }'
```

```json
{
  "data": {
    "record_type": "provider_key",
    "id": "2d4f6a8b-1c3e-4a5b-9d7f-0e1a2b3c4d5e",
    "name": "anthropic-production",
    "provider": "anthropic",
    "version": 1,
    "created_at": "2026-09-22T10:00:00Z",
    "updated_at": "2026-09-22T10:00:00Z"
  }
}
```

Never keep a logged copy of the request that contains the secret. `GET /provider_keys` lists provider keys and `GET /provider_keys/{id}` reads one; neither shows the secret.

Reference the provider key in the group's `provider_key_ids`, and include the BYOK models you want in `allowed_models`. A group holds at most one key per provider, the key's provider must match at least one BYOK model the group allows, and the key and group must be in the same account.

Set both when creating a group, or update an existing group with an ETag-protected `PATCH`. `allowed_models` is replaced as a whole, so include every model the group should keep:

```bash
curl -X PATCH "https://api.telnyx.com/v2/llm_token_gateway/token_groups/$GROUP_ID" \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "If-Match: $GROUP_ETAG" \
  -d '{
    "allowed_models": ["Kimi-K3", "claude-sonnet-5"],
    "provider_key_ids": ["2d4f6a8b-1c3e-4a5b-9d7f-0e1a2b3c4d5e"]
  }'
```

Applications keep using their `ltg_sk_` token key exactly as before. Never place the provider secret in SDK configuration or application code. Call BYOK models on `/v1/chat/completions`; Anthropic BYOK models also work on `/v1/messages` with the [Anthropic SDK](/docs/inference/ai-gateway/inference-api#anthropic-sdk).

A BYOK model works only when the calling key's group has an attached provider key for that model's provider. Without one, requests fail with `503` and code `enforcement_unavailable`. They never fall back to a Telnyx-hosted model.

## Billing and reporting

- Requests on BYOK models are billed by your provider on your provider account, not by Telnyx.
- AI Gateway budgets and rate limits still apply.
- Usage reporting records every BYOK request. Its `cost` is the [budget reference valuation](/docs/inference/ai-gateway/controls#budgets-and-billing), not a Telnyx charge.

## Rotate a provider secret

Provider keys cannot be edited; `PATCH` returns `405`. To change a secret, create a new provider key, attach it to each group in place of the old one, then delete the old provider key.

## Delete a provider key

`DELETE /provider_keys/{id}` requires the provider key's current `ETag` in `If-Match`. Read the key first to get it:

```bash
PROVIDER_KEY_ETAG=$(curl -sS -I "https://api.telnyx.com/v2/llm_token_gateway/provider_keys/$PROVIDER_KEY_ID" \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  | awk 'tolower($1) == "etag:" { sub(/\r$/, "", $2); print $2 }')

curl -X DELETE "https://api.telnyx.com/v2/llm_token_gateway/provider_keys/$PROVIDER_KEY_ID" \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "If-Match: $PROVIDER_KEY_ETAG"
```

Deleting a provider key detaches it from every group that references it. Requests already in progress complete; new requests on that provider's models from those groups fail with `503` until another key for the provider is attached.

---

### Usage reporting

> Source: https://developers.telnyx.com/docs/inference/ai-gateway/usage.md

AI Gateway reports request usage by token group, user, key and asserted end user. Choose the endpoint for the report:

| Endpoint | Returns |
| --- | --- |
| `GET /usage/summary` | Complete group totals, daily and model breakdowns, and guardrail counts in one response. |
| `GET /spend/events` | One row per request. |
| `GET /spend/summary` | Totals grouped by one dimension. |
| `GET /usage/export` | Replayable, content-free usage revisions and guardrail events for an external collector. |

All paths are relative to `https://api.telnyx.com/v2/llm_token_gateway` and authenticate with the Telnyx account API key.

## Group usage report

Use [Get gateway usage summary](/api-reference/ai-gateway/get-gateway-usage-summary) to build a group dashboard without fetching individual request pages. Each response uses one database snapshot and reflects the latest available usage corrections.

The credential needs the `llm_token_gateway.usage.read` permission for this combined report. Existing spend and guardrail permissions do not grant access to it.

The endpoint requires `token_group_id`, `start_date` and `end_date` and accepts optional `metadata[<key>]` filters. The group must belong to the authenticated account. Dates are UTC `YYYY-MM-DD`, with an inclusive start and exclusive end, spanning 1 to 31 days. Pagination parameters, `group_by` and other filters are not accepted. Tag filters apply to every report section and are echoed in `meta.metadata`.

```bash
curl -G https://api.telnyx.com/v2/llm_token_gateway/usage/summary \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  --data-urlencode "token_group_id=$GROUP_ID" \
  --data-urlencode "start_date=2026-09-01" \
  --data-urlencode "end_date=2026-09-30"
```

| Response field | Meaning |
| --- | --- |
| `data.totals` | Metrics for every matching request. |
| `data.by_day` | The same metrics with a `date` field for each UTC day, including days with zero activity. |
| `data.by_model` | The same metrics with a `model` field, ordered by request count descending, then model name. |
| `data.guardrails.blocked_events`, `data.guardrails.flagged_events` | Complete event counts for the group and range. These count events, not distinct requests. |
| `data.guardrails.recent_events` | Up to 20 newest [guardrail events](/docs/inference/ai-gateway/guardrails#query-findings). Their count does not limit the totals. |
| `meta` | The requested `token_group_id`, `start_date` and `end_date`. |

Each metric row contains `requests`, `input_tokens`, `output_tokens`, `spend`, `reserved_spend`, `unknown_requests`, `cache_hits`, `succeeded_requests`, `failed_requests` and `partial_requests`. Known input and output tokens are counted independently, including when the other token category remains unknown. Unknown cost is excluded from `spend`; `reserved_spend` reports the unresolved budget hold in USD.

Totals, daily rows and model rows are alternative views of the same requests. Do not add the sections together. The report includes up to 1,000 model rows and returns HTTP 503 if the report cannot be completed within its limits; it never returns a silently truncated success. A failed report must not be displayed as zero usage.

The spend endpoints below retain their separate filters and snapshot pagination.

## Date range and filters

| Parameter | Notes |
| --- | --- |
| `start_date` | Inclusive UTC date, `YYYY-MM-DD`. Required for `/spend/summary`. |
| `end_date` | Exclusive UTC date, `YYYY-MM-DD`. Required for `/spend/summary`. |
| `token_group_id`, `token_user_id`, `token_key_id` | Optional UUID filters. |
| `end_user_id` | Optional end-user identifier filter. |
| `group_by` | `/spend/summary` only, required: `token_group`, `token_user`, `token_key`, `end_user` or `metadata:<key>`. |
| `page[number]`, `page[size]`, `page[snapshot]` | Snapshot pagination; see below. |

The range is half-open, `[start_date, end_date)`, and spans at most 31 days. Send ISO dates, not timestamps. If you omit both dates on `/spend/events`, it returns the previous full UTC day. To include today, set `end_date` to tomorrow's date.

## Spend events

```bash
curl --globoff -G https://api.telnyx.com/v2/llm_token_gateway/spend/events \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  --data-urlencode "start_date=2026-09-22" \
  --data-urlencode "end_date=2026-09-23" \
  --data-urlencode "token_group_id=$GROUP_ID" \
  --data-urlencode "page[size]=100"
```

Each event carries:

| Field | Meaning |
| --- | --- |
| `id`, `request_id`, `created_at` | Event identity and the `X-Request-ID` of the inference request. |
| `token_group_id`, `token_user_id`, `token_key_id`, `end_user_id` | Attribution. `token_user_id` is null for service keys; `end_user_id` is null when the request asserted none. |
| `requested_model`, `model`, `attempts` | Originally requested model, final serving/attempted model and provider attempt count. Tokens and cost include all attempts. |
| `metadata` | Captured request tags, or `{}` when untagged. |
| `input_tokens`, `output_tokens` | Null when usage is unknown. |
| `cost` | USD at the flat reference rate used for budgets; see [Budgets and billing](/docs/inference/ai-gateway/controls#budgets-and-billing). Null when usage is unknown. |
| `status` | `succeeded`, `failed`, `partial` or `unknown`. |
| `usage_status` | `known`, `unknown` or `reconciled` after a later correction. |
| `configuration_version`, `rate_version` | The policy and rate snapshot the request was valued under. |
| `reservation_micro_usd` | The reservation held for the request in micro-USD. It stays outstanding while usage is unknown. |

## Spend summary

`group_by` selects the dimension. Each row totals the requests attributed to one value of that dimension within the range and filters.

```bash
curl --globoff -G https://api.telnyx.com/v2/llm_token_gateway/spend/summary \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  --data-urlencode "start_date=2026-09-01" \
  --data-urlencode "end_date=2026-10-01" \
  --data-urlencode "group_by=token_key"
```

```json
{
  "data": [
    {
      "dimension_id": "9b7e4a10-3c2d-4f5e-8a6b-1d2c3e4f5a60",
      "name": "support-backend",
      "token_group_id": "5f1c9d2e-7b3a-4c8e-9f21-0a6d4e8b1c33",
      "token_user_id": null,
      "token_key_id": "9b7e4a10-3c2d-4f5e-8a6b-1d2c3e4f5a60",
      "end_user_id": null,
      "spend": 1.204311,
      "requests": 4180,
      "input_tokens": 912340,
      "output_tokens": 401277,
      "unknown_requests": 2,
      "reserved_spend": 0.0125
    }
  ],
  "meta": {
    "page_number": 1,
    "page_size": 100,
    "has_more": false,
    "snapshot": "...",
    "start_date": "2026-09-01",
    "end_date": "2026-10-01",
    "group_by": "token_key"
  }
}
```

The `group_by` dimensions are alternative views of the **same** requests. Do not add totals from different dimensions together.

## Request tags

Attach up to five attribution tags to an inference request using the `x-ltg-metadata` header. Send one JSON object, at most 2,048 ASCII bytes; encode non-ASCII characters as JSON Unicode escapes.

```text
x-ltg-metadata: {"team":"support","environment":"production","experiment":2,"priority":true}
```

Keys are case-sensitive, 1–64 characters, and match `[A-Za-z0-9][A-Za-z0-9_.-]*`. Prefixes `ltg_`, `telnyx_`, `litellm_` and `cf_` are reserved regardless of case. Values are non-empty strings of at most 256 characters without control/format characters, integers from -9,007,199,254,740,991 to 9,007,199,254,740,991, or booleans. Floats, null, arrays, objects and credential-like values are rejected with `400 invalid_metadata`; the error never echoes the header. Use non-sensitive attribution labels.

Tags do not configure budgets or routing. They are never forwarded upstream, do not affect cache identity and are not echoed by inference responses. They appear on spend events, guardrail events and usage-summary recent events, including the current request's tags on cache hits. They are separate from the Anthropic body field `metadata.user_id`; OpenAI body `metadata` is not accepted.

Filter `/spend/events`, `/spend/summary`, `/guardrail_events` or `/usage/summary` with `metadata[team]=support`. At most five filters are combined with AND and match exact text: booleans use `true`/`false`, integers decimal text. Consequently a filter value `true` matches both boolean true and the string `"true"`. Keep the same filters when paging a snapshot.

Use `/spend/summary?group_by=metadata:team` to group by a tag's text value. Requests missing the tag form a row with `dimension_id: null`. Tags do not create portal-wide defaults: applications supply them on each request.

## Durable usage export

[Export gateway usage](/api-reference/ai-gateway/export-gateway-usage) returns content-free `inference.request` and `guardrail.evaluation` records. It requires the separate `llm_token_gateway.usage_export.read` permission; spend/report permissions do not grant it. This is a pull API; no push destination is configured on a gateway.

| Parameter | Meaning |
| --- | --- |
| `limit` | 1–1,000 records; default 500. |
| `start_time` | First page only, RFC 3339 timestamp with an offset, at most 30 days ago and not in the future. Defaults to 24 hours ago. |
| `cursor` | Resume from `meta.next_cursor`; cannot be combined with `start_time`. |
| `types` | Comma-separated subset of `inference.request,guardrail.evaluation`; defaults to both and must stay the same for a cursor. |

Persist records in your sink before saving `meta.next_cursor`. Deduplicate on record `id`: delivery is at least once. A corrected inference record has a higher `revision` and a `supersedes` link; retain the highest revision per `request_id`. Join guardrail records on `request_id`. Both record types include captured request tags in `metadata`, or `{}` for an untagged request. Persist these tags with the record for attribution.

While `meta.has_more` is true, fetch the next page immediately. When false, keep the cursor and poll later; an empty page does not end the feed. `meta.horizon_lag_seconds` reports delayed visibility of committed records. Retention is guaranteed for 30 days.

On `409 cursor_expired`, restart from a `start_time` within the retained window and deduplicate. A notified logical restore also requires this restart even if an old cursor is still accepted. On `503 enforcement_unavailable`, preserve the cursor and honor `Retry-After`; the endpoint returns an error instead of partial or truncated data. Unknown usage and reference spend retain the same caveats as other reports; this export is not invoice truth.

## Report limits

Each spend report includes at most 10,000 matching requests, regardless of the requested page size. A report that exceeds this limit returns HTTP 503 instead of partial totals. Reduce the date range or narrow the resource filters before retrying; reducing the page size does not reduce the work needed to create the report snapshot.

Treat a failed or timed-out report as unavailable; do not replace its totals with zero.

## Pagination

The first page returns `meta.snapshot`, a stable view bound to the account, range and filters for 15 minutes. While `meta.has_more` is true, request the next `page[number]` with the same parameters plus `page[snapshot]=<meta.snapshot>`. Changing filters or mixing snapshots returns `409`. Treat an export as complete only once `has_more` is false.

On a snapshot `409`, correct the parameters to match the original valid snapshot, or restart with `page[number]=1` and omit `page[snapshot]` to obtain a new snapshot. The response carries `Retry-After`, but waiting does not make an invalid or expired snapshot usable. When restarting, discard the earlier pages; do not combine pages from different snapshots in one export.

## Interpreting the numbers

- **Unknown usage is not zero.** A request whose usage never arrived, for example an interrupted stream, keeps its reservation as `reserved_spend` and reports `cost: null` with `usage_status: "unknown"`. Later reconciliation updates the same event to `reconciled`.
- **Usage is billed at standard pricing.** Usage of Telnyx-hosted models is billed to your Telnyx account at standard [Telnyx AI Inference pricing](https://telnyx.com/pricing/inference-api) for each model.
- **BYOK requests are reported like any other.** Requests on [bring-your-own-key models](/docs/inference/ai-gateway/byok) appear in spend events and summaries like any other request. Their `cost` is the budget reference valuation, not a Telnyx charge; your provider bills them.
- **Spend is not an invoice.** `cost` and `spend` are measured at a flat reference rate (USD 5 per million input tokens, USD 15 per million output tokens) for enforcing budgets and attribution. They are not your invoice.
- **Revoked and deleted resources keep their history.** Filters by ID continue to work after a key, user or group is deleted.

---

### Errors

> Source: https://developers.telnyx.com/docs/inference/ai-gateway/errors.md

Both planes return a structured `errors` array. The inference plane additionally wraps it in the envelope the calling SDK expects, so OpenAI and Anthropic SDK exceptions work unchanged while the Telnyx detail remains available.

## Error envelopes

```json Management
{
  "errors": [
    {
      "code": "precondition_failed",
      "title": "Stale If-Match",
      "detail": "The resource version has changed; read it again.",
      "meta": { "current_version": 4 }
    }
  ]
}
```

```json OpenAI-compatible
{
  "error": {
    "message": "Send either max_tokens or max_completion_tokens, not both.",
    "type": "invalid_request_error",
    "param": "max_completion_tokens",
    "code": "invalid_request"
  },
  "errors": [
    {
      "code": "invalid_request",
      "title": "Invalid request",
      "detail": "Send either max_tokens or max_completion_tokens, not both.",
      "meta": {}
    }
  ]
}
```

```json Anthropic-compatible
{
  "type": "error",
  "error": {
    "type": "permission_error",
    "message": "Model is not available to this token key."
  },
  "request_id": "0b1c2d3e-4f50-4617-8a29-3b4c5d6e7f80",
  "errors": [
    {
      "code": "model_not_in_catalog",
      "title": "Model not allowed",
      "detail": "Model is not available to this token key.",
      "meta": { "scope": "token_key" }
    }
  ]
}
```

Every response carries an `X-Request-ID` header. Keep it, together with the `code`, when reporting a problem. Do not log authorization headers, token keys or full request objects.

An error that occurs after a streaming response has started cannot change the HTTP status. On the OpenAI surface it arrives as an error chunk; on the Anthropic surface, as an `event: error` frame. Consume every stream to completion and handle the SDK's stream exceptions. A buffered-response delivery failure after JSON headers have been sent aborts the incomplete HTTP response; treat it as a transport failure with potentially completed provider work.

## Additional request errors

| HTTP status | Code | Handling |
| --- | --- | --- |
| `400` | `invalid_metadata` | Correct the request-tag header; the error does not echo it. |
| `400` | `request_exceeds_tpm_limit` | The reservation exceeds an entire key, user or group TPM limit. Set an explicit output limit, reduce input or planned attempts, or ask the policy owner to adjust the limit. No provider dispatch or `Retry-After`; do not retry unchanged. |
| `400` | `context_length_exceeded` | Reduce input and/or requested output. No retry or fallback is attempted for this context rejection. |
| `413` | `request_too_large` | Reduce the serialized JSON body to the model's advertised byte envelope. |
| `429` | `concurrency_limit_exceeded` | The key or group is at its concurrent-request cap. Inspect `errors[0].meta.scope` and retry with bounded backoff after `Retry-After`. |
| `504` | `upstream_timeout` | A provider attempt or the server-side request lifetime timed out before the client response started. Buffered requests can time out while reading the provider body. Usage may be unknown; a new request can incur additional work. |
| `409` | `cursor_expired` | Restart usage export within its retained time window and deduplicate by record ID. Waiting does not repair the cursor. |

## Pending provisioning

An inference response with HTTP `503`, code `enforcement_unavailable` and the optional header `x-ltg-enforcement-reason: provisioning` indicates that the request encountered pending provisioning of the token key or its group/user policy. The header does not change the status or error body, and does not guarantee when provisioning will complete. If the header is absent, the provisioning state is unspecified.

Retry [model discovery](/docs/inference/ai-gateway/inference-api#model-discovery) with bounded backoff, respecting `Retry-After`, before sending the first inference request. Stop and surface the failure if provisioning exceeds your application's deadline. Do not repeatedly create replacement keys or automatically replay inference requests.

## Status codes

| Status | Meaning | Action |
| --- | --- | --- |
| `400` | A guardrail policy blocked the prompt or response; invalid body, unsupported option or model-specific option, unknown field, request over the size limits, token key limit out of range, invalid date or page parameter. | Correct the request. |
| `401` | Missing or wrong credential for this plane. | Use a Telnyx API key on the management plane and an `ltg_sk_` token key on the inference plane. |
| `403` | Blocked, revoked or expired key; blocked resource; model not allowed; budget or end-user policy denied. | Read the `code`. Do not treat `403` as retryable. |
| `404` | Resource does not exist or belongs to another account. | Check the ID. |
| `405` | Method not allowed, for example `PATCH` on a provider key. | Provider keys cannot be edited; create a new one instead. |
| `409` | Idempotency conflict, membership conflict or pagination snapshot conflict. | Read the `code`. |
| `412` | Stale `If-Match`. | Read the resource and retry with the current ETag. |
| `428` | Missing `If-Match` (or `If-None-Match: *` on end-user create). | Add the precondition header. |
| `429` | Rate limit exceeded. | Wait for `Retry-After`. Remember that inference retries are not idempotent. |
| `502` | The model provider failed. | Fail closed. The request may or may not have consumed provider work. |
| `503` | Enforcement, catalog, policy propagation or a required dependency is unavailable; a BYOK model lacks an attached provider key; or a spend report exceeds its capacity limit. | Fail closed. For an oversized spend report, narrow the dates or resource filters before retrying. Do not switch credentials or bypass the gateway. |

## Error codes

| Code | Status | Meaning |
| --- | --- | --- |
| `invalid_request` | 400 | Malformed or unsupported request, including a request over the [size limits](/docs/inference/ai-gateway/inference-api#request-size-limits) and a request to `/v1/messages` with a model that is not an Anthropic BYOK model (use `/v1/chat/completions` for those). |
| `prompt_blocked` | 400 | The group guardrail policy blocked the prompt before provider dispatch or budget reservation. |
| `response_blocked` | 400, or SSE error after HTTP 200 | The group guardrail policy withheld the generated response. Provider usage still applies. See [Guardrails](/docs/inference/ai-gateway/guardrails#blocking-and-usage). |
| `limit_out_of_range` | 400 | A token key `max_budget`, `rpm_limit` or `tpm_limit` is 0 or above its maximum. See [Token key limits](/docs/inference/ai-gateway/controls#token-key-limits). |
| `invalid_token_key` | 401 | The inference credential is missing, malformed or unknown. |
| `unauthorized` | 401 / 403 | The credential is not valid for this plane or resource. |
| `token_key_blocked` | 403 | The key is blocked, revoked or expired. |
| `resource_blocked` | 403 | The key's user, group or asserted end user is blocked. |
| `budget_exceeded` | 403 | A key, user or group budget lacks headroom for the request's reservation. |
| `end_user_budget_exceeded` | 403 | The asserted end user's budget lacks headroom. |
| `model_not_in_catalog` | 403 | The model is not in the key's or group's allowlist. |
| `rate_limit_exceeded` | 429 | An RPM or TPM limit was hit. |
| `not_found` | 404 | No such resource in this account. |
| `conflict` | 409 | Membership or pagination snapshot conflict. |
| `idempotency_conflict` | 409 | The `Idempotency-Key` was reused with a different body, or the original request is still in progress. |
| `precondition_failed` | 412 | `If-Match` does not match the current version. |
| `precondition_required` | 428 | A required precondition header is missing. |
| `enforcement_unavailable` | 503 | Policy, catalog or a required dependency is unavailable; a [BYOK model](/docs/inference/ai-gateway/byok) lacks an attached provider key; or a spend report exceeds the [report limits](/docs/inference/ai-gateway/usage#report-limits). |
| `upstream_error` | 502 | The model provider returned an error. |

## Handling guidance

- **Never resolve an error by escalating credentials.** A Telnyx API key, provider secret or any other credential is rejected on the inference plane by design.
- **Retry management mutations with the same idempotency key.** A new key on retry can create a duplicate resource.
- **Do not automatically retry inference.** A timeout or `502` is not proof that no provider work happened. Retry only errors that occurred before dispatch, and honor `Retry-After` on `429`.
- **Re-read before re-writing.** On `412`, fetch the resource, review the change that landed, and apply your update to the current version.
- **Fail closed on `503`.** For inference, the old policy may no longer apply; do not assume the request is authorized. For a BYOK model, check that the group has a provider key attached for that model's provider.
- **Narrow oversized spend reports before retrying.** Reports exceeding 10,000 matching requests return `503` with `enforcement_unavailable`. Reduce the date range or narrow resource filters; reducing page size or retrying the unchanged query does not resolve this limit. Treat a failed report as unavailable, not zero usage.

---

## API Reference (AI Gateway)

### AI Gateway

- [List gateway spend events](https://developers.telnyx.com/api-reference/ai-gateway/list-gateway-spend-events.md): Return request-level usage history for the authenticated account. Supply both UTC dates or neither; omitting both returns the previous full UTC day. Dates use…
- [Get gateway spend summary](https://developers.telnyx.com/api-reference/ai-gateway/get-gateway-spend-summary.md): Return request counts, token counts and reference spend grouped by one attribution dimension. Both UTC dates and group_by are required. Dates use a half-open r…
- [Get gateway usage summary](https://developers.telnyx.com/api-reference/ai-gateway/get-gateway-usage-summary.md): Gateway reference/enforcement usage, not invoice truth or BYOK provider charges. Requires the dedicated llm_token_gateway.usage.read V2 action for combined spe…
- [Export gateway usage](https://developers.telnyx.com/api-reference/ai-gateway/export-gateway-usage.md): Durable, replayable, content-free usage export for customer pipelines. Requires the dedicated llm_token_gateway.usage_export.read V2 action; spend.read and usa…
- [List gateway token groups](https://developers.telnyx.com/api-reference/ai-gateway/list-gateway-token-groups.md): List gateway token groups within the authenticated Telnyx account. Foreign and unknown resource IDs return the same 404 response.
- [Create gateway token group](https://developers.telnyx.com/api-reference/ai-gateway/create-gateway-token-group.md): Create gateway token group within the authenticated Telnyx account. Foreign and unknown resource IDs return the same 404 response.
- [Get gateway token group](https://developers.telnyx.com/api-reference/ai-gateway/get-gateway-token-group.md): Get gateway token group within the authenticated Telnyx account. Foreign and unknown resource IDs return the same 404 response.
- [Update gateway token group](https://developers.telnyx.com/api-reference/ai-gateway/update-gateway-token-group.md): Update gateway token group within the authenticated Telnyx account. Foreign and unknown resource IDs return the same 404 response. Omitted fields are preserved…
- [Delete gateway token group](https://developers.telnyx.com/api-reference/ai-gateway/delete-gateway-token-group.md): Revoke resource and its keys atomically; retain historical attribution, never detach into an ungoverned service key.
- [List gateway token keys](https://developers.telnyx.com/api-reference/ai-gateway/list-gateway-token-keys.md): List gateway token keys within the authenticated Telnyx account. Foreign and unknown resource IDs return the same 404 response.
- [Create gateway token key](https://developers.telnyx.com/api-reference/ai-gateway/create-gateway-token-key.md): Create gateway token key within the authenticated Telnyx account. Foreign and unknown resource IDs return the same 404 response. The token secret is returned o…
- [Get gateway token key](https://developers.telnyx.com/api-reference/ai-gateway/get-gateway-token-key.md): Get gateway token key within the authenticated Telnyx account. Foreign and unknown resource IDs return the same 404 response.
- [Update gateway token key](https://developers.telnyx.com/api-reference/ai-gateway/update-gateway-token-key.md): Update gateway token key within the authenticated Telnyx account. Foreign and unknown resource IDs return the same 404 response. Omitted fields are preserved.…
- [Delete gateway token key](https://developers.telnyx.com/api-reference/ai-gateway/delete-gateway-token-key.md): Revoke this token key. Acknowledged revocation blocks new admissions with the key; requests already admitted may complete. Historical usage attribution is reta…