Skip to main content
Every inference request passes an admission check before it is dispatched to a model. Admission evaluates the token key, its user, its group and the asserted end user together; the request is denied if any scope fails. This page describes each control and its observable behavior.

​
Enforcement scopes

Model access is the intersection of the group’s allowed_models and the key’s allowed_models (when set). A blocked key, user, group or end user denies the request with 403.

​
Budgets

Budgets are USD amounts with at most six decimal places, enforced independently at the key, user, group and end-user scopes.
  • On groups, users and end users, max_budget: null is uncapped and max_budget: 0 denies every paid request. Token keys differ; see Token key limits.
  • budget_duration of 1d, 7d or 30d starts an anchored period at the moment the budget is committed. Periods roll from that anchor, not from midnight or the calendar month. A null duration is a lifetime budget.
  • Changing the amount keeps the current period. Changing the duration starts a new anchored period. Neither change erases history.
  • The response fields spend, reserved_spend, budget_started_at and resets_at on each resource show the current period.

​
Reservations

Before a request is dispatched, the gateway reserves a conservative upper bound on its cost from the input size and the resolved output allowance. Explicit max_tokens or max_completion_tokens bounds that allowance. Hosted models that leave an omitted or null limit to the engine reserve their full context window as output; models with an 8,192 default reserve that default. See Output limits and defaults. If any scope lacks that much headroom, the request is denied with 403 and code budget_exceeded (or end_user_budget_exceeded). The reservation is not shrunk to fit. Multiple inference requests can use the same token key concurrently. Concurrent cache misses each reserve their own cost bound and must pass the applicable budget and rate limits. A request with unknown usage retains its monetary reservation without blocking later requests that fit within the remaining limits. After the response completes, the reservation is replaced by the actual cost. If the outcome is unknown, for example because the stream was interrupted before usage was reported, the reservation is retained as exposure and the spend event reports cost: null. Unknown usage is not zero usage, and a timeout is not a refund. Budgets are an application control, not an absolute guarantee of spend.

​
Budgets and billing

  • Usage of Telnyx-hosted models is billed to your Telnyx account at standard Telnyx AI Inference pricing for each model.
  • Requests on bring-your-own-key models are billed by your provider on your provider account, not by Telnyx. Budgets and rate limits still apply and act as a guard on that provider spend.
  • Budgets and the cost values in usage reporting are measured at a flat reference rate of USD 5 per million input tokens and USD 15 per million output tokens, for enforcing limits and attribution. They are not your invoice.

​
Rate limits

rpm_limit and tpm_limit are evaluated over rolling 60-second windows, not calendar minutes, at the key, user and group scopes.
  • On groups and users, null removes the limit and zero denies every request. Token keys differ; see Token key limits.
  • Temporary rate contention returns 429 with a Retry-After header.
  • A single request whose reservation exceeds an entire configured TPM limit returns 400 request_exceeds_tpm_limit without Retry-After, before provider dispatch. Set an explicit output limit, reduce input or planned attempts, or ask the policy owner to adjust the limit. Retrying the unchanged request cannot succeed.
  • tpm_limit counts reserved input and output tokens. An omitted or null hosted output limit can reserve the full context window as output. Set max_tokens or max_completion_tokens explicitly to bound the reservation; the input allowance must also fit within tpm_limit.

​
Concurrent requests

Set max_parallel_requests on a token group or token key to an integer from 1 to 1,000. Null (the default) means no concurrency cap at that scope; omission in a PATCH preserves the current value and null clears it. Both limits apply. The group limit covers all its keys across gateway replicas. Token users and end users do not accept this field. A slot covers an admitted cache miss until settlement or release; cache hits bypass the cap. Lowering a limit does not cancel requests already admitted. New requests above the limit receive 429 concurrency_limit_exceeded, errors[0].meta.scope (token_key or token_group) and Retry-After: 1. No provider call, budget reservation or rate allocation is made for a denied request. The retry delay is advisory, not a promised completion time. Unknown usage keeps its monetary hold without a concurrency slot. After a crashed owner, a slot can remain occupied until recovery or lease expiry (up to five minutes). This control bounds gateway-owned requests, not remote provider activity after a lost connection.

​
Fallback models and retries

Configure fallbacks and retries on the token group. Both apply to Telnyx-hosted routes and default to disabled. BYOK requests do not use them.
This is a partial group PATCH body; use the current ETag and an idempotency key as described under Mutation safety. max_attempts × (1 + longest fallback chain) must not exceed eight. Invalid configuration, including removing an allowed model still referenced by a chain, returns 400. PATCH omission preserves either policy; retries: null disables retries. The gateway retries the same model only when it definitely did no provider work: a connection failure before dispatch, or provider 429/529. An uncertain failure such as a timeout or 408/500/502/503/504 moves to the next fallback instead. Retry waits follow the selected curve, are capped at five seconds and jittered between half and the full delay. A provider Retry-After of at most five seconds is honored; a longer delay moves to the next step. The request deadline still applies. Fallback destinations are checked against their capabilities and request-size limits. A context-length rejection is terminal. Once the first provider event is forwarded, the model never changes and no retry or fallback occurs. If every attempt fails before that event, the response is HTTP 502 upstream_error, or 504 upstream_timeout when the last attempt timed out. An error after streaming begins is an error event followed by closure, without a success terminator. The response model and x-ltg-served-model identify the serving model; x-ltg-attempts counts provider attempts (zero for a cache hit). Spend events include requested_model, the serving model and attempts. Tokens and spend include all attempts. Budget and TPM capacity for planned attempts are reserved up front; unused reservations are released at settlement. Unknown usage retains its monetary hold even if a later attempt succeeds. SDK retries are additional requests with their own attempts and reservations. Set SDK retries to zero when relying on gateway retries. Request headers can tighten the group policy; see Retries and timeouts.

​
Response caching

Set group cache_ttl to 1–86,400 seconds to enable response caching, or null to disable it. Omission in a PATCH preserves the current setting. Caching is best effort; entries can be evicted before their TTL, and equivalent requests on different replicas can miss. Only complete supported responses are cached. Partial streams, failures and responses that cannot be reconstructed safely are excluded. Cache identity includes the applicable key, group, user, model, provider route and policy. A policy change prevents reuse under the old policy. Cache hits avoid provider work, report zero reference cost and provider tokens, and do not consume concurrency slots. Authorization and policy checks still apply. Send "cache": {"no-cache": true} in the inference request to bypass caching. Request metadata tags are not part of cache identity; a hit is attributed to the tags on the current request. Caching does not configure a provider’s prompt cache.

​
Token key limits

Token keys always have budget and rate caps. The optional concurrency cap is separate. Group and user budget/rate limits are unchanged: null is uncapped and zero denies.
  • An omitted or null key limit is stored as its maximum. On a blocked key, a null limit is kept until the key is unblocked.
  • The default USD 1,000 budget is a lifetime budget unless budget_duration is set.
  • A value of 0 or above the maximum returns 400 with code limit_out_of_range.
  • A negative, fractional or non-numeric value, or a budget with more than six decimal places, returns 400 with code invalid_request.
  • To deny a key, block it (blocked: true) or revoke it.

​
End-user identity

The OpenAI user field and the Anthropic Messages metadata.user_id field identify an account-scoped end user in the same namespace. The value is used for end-user budgets and blocks and appears as end_user_id in usage reporting. The identifier is an assertion by whoever holds the token key, not an authenticated identity. A key embedded in a client can assert any value. Bind end-user identifiers to authenticated sessions in a trusted backend, and set required_end_user_id: true on a key when every request must carry one. That flag checks presence only.

​
Mutation safety

Management mutations are protected by idempotency keys and ETag preconditions; see Conventions. A committed policy change applies to new admissions once it has propagated. During propagation, inference can return 503 with code enforcement_unavailable. Fail closed in that case rather than retrying under the assumption that the previous policy still applies.

​
Content guardrails

Set a group-level guardrails policy to flag or block supported secrets and sensitive data before provider dispatch or before delivering a response. Prompt blocks reserve no budget; response blocks still account for provider work. See Guardrails for configuration and streaming behavior.