Routing and Fallbacks

Beta. An instance sets no routing of its own. Each request sets its own fallbacks, retries, and timeout, as on this page.

A request can retry when it fails, limit how long each attempt takes, and fall back to other models.

Retries

An instance retries a failed request only when the request sets num_retries. The retries run before the request fails or falls back. A model that keeps failing is taken out of rotation for 5 seconds, and requests for it return 429 meanwhile.

Fallbacks

When a request fails and its retries are spent, the gateway tries the request’s fallbacks in order. When every fallback fails, the response is the first model’s error.

Per-Request Settings

A request can set its own routing in the request body:

Field	What It Does
`fallbacks`	The models to try, in order, if this request fails. An entry can be a model name, or an object with `model` and parameters that replace the request’s own for that model.
`disable_fallbacks`	`true` skips the request’s fallbacks.
`num_retries`	How many times to retry the request.
`timeout`	How long each attempt may take, in seconds. Without it, an attempt that is not streamed fails after 30 seconds.
`stream_timeout`	The same, for a streamed request.

A fallback model must be on the key’s allow list, or the gateway refuses the request.

Fallback Example

This request asks GLM-5.3-Flash for more output tokens than it allows. Its fallback, Qwen3.8-27B-nothink, replaces max_tokens with 8:

Host

curl "$GATEWAY_URL/v1/chat/completions" \
  -H "Authorization: Bearer $AGW_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "GLM-5.3-Flash",
    "max_tokens": 100000000,
    "messages": [{ "role": "user", "content": "Say hi" }],
    "fallbacks": [{ "model": "Qwen3.8-27B-nothink", "max_tokens": 8 }]
  }'

GLM-5.3-Flash refused the request with 422, and the fallback answered. After a fallback, model holds the Telnyx inference model that answered:

Output

{
  "id": "chatcmpl-10acb933-e5c5-42aa-aea4-82549133320a",
  "model": "Qwen/Qwen3.8-27B",
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "length",
      "message": { "content": "Hi! How can I help you today", "role": "assistant", ... },
      ...
    }
  ],
  "usage": { "completion_tokens": 8, "prompt_tokens": 14, "total_tokens": 22, ... },
  ...
}

Without fallbacks, or with "disable_fallbacks": true, the same request returns the 422.

Get Started

Targets

Virtual Private Cloud

Outbound Authentication

Inbound Authentication

Authorization and Policy

Traffic Control

Audit and Observability

API Reference

For AI Agents

Routing and Fallbacks

Retries

Fallbacks

Per-Request Settings

Fallback Example

​Retries

​Fallbacks

​Per-Request Settings

​Fallback Example

Retries

Fallbacks

Per-Request Settings

Fallback Example