Voice AI

Failover You Can Prove: Policy-Driven Model Fallbacks with Telnyx AI Gateway

How to provision a token group with fallbacks and retries, trip the primary on purpose, and verify the failover in both the response headers and the spend ledger.

At 2 a.m., a patient on a telehealth line describes chest tightness. The after-hours triage assistant drafts a summary of the intake call, and that summary has to be in front of a clinician in seconds — because in this scenario, a stalled model call is not an inconvenience. It's a patient-safety event and a compliance exposure under the network's clinical documentation policy.

Engineers can't control which hosted model has a bad night. What they can control is what happens when one does — and whether they can prove what happened afterward. That second part is where most stacks fall apart: a client-side retry loop quietly papers over the failure, the dashboard shows a 200, and the evidence of which model actually served the request is gone.

The gateway-fallbacks sample exists to close that gap. It provisions a Telnyx AI Gateway token group with policy-driven fallbacks and retries, trips the primary model on purpose, and proves the failover twice: once on the live path, in the response headers, and once on the record, in the spend ledger. Failover you cannot observe is failover you cannot trust. Here's how it works — and how to run it yourself.

What the App Does

The sample is a resilience desk: a small Python codebase that exercises the full lifecycle of policy-driven failover, end to end.

On the management plane, it provisions a token group that names an allowed model set, declares which model takes over when the primary stalls, sets retry behavior, and attaches a budget. It then issues a token key scoped to that group — the secret is returned only on the create response, so the sample stores it immediately. Model discovery (GET /v1/models) is scoped to the calling token key, so the app only ever sees what its group allows.

On the inference plane, it sends Chat Completions through an OpenAI-compatible endpoint using only that token key. Then comes the interesting part: the forced-failure run reuses the provisioned group and key and sends a streaming request against the primary with a per-request timeout overlay of one second. The primary's first token misses the deadline, and the gateway hands the request to the fallback.

Finally, the observation layer proves what happened from two independent directions:

  • The live path: every response carries x-ltg-served-model (the model that actually answered) and x-ltg-attempts (how many attempts it took).
  • The ledger: GET /spend/events records requested_model, model, attempts, status, and cache_hit side by side on one row.

After every call, the sample also fetches the token group and prints the raw budget and reservation fields — spend, reserved_spend, max_budget, budget_duration — so you can watch capacity being reserved and released. And when it's done, it cleans up after itself with ETag-guarded deletes.

How It Works

Two planes and one observation layer

The boundary between the planes is where most gateway confusion lives, so the sample makes it explicit:

   MANAGEMENT PLANE              INFERENCE PLANE
   api.telnyx.com/v2             llm.telnyx.com/v1
   auth: TELNYX_API_KEY          auth: token key ONLY
   (Telnyx Python SDK)           (OpenAI SDK, max_retries=0)

   POST   /token_groups   ──►    GET  /models
   POST   /token_keys    ──►    POST /chat/completions
   GET    /spend_events          gateway owns retries + fallbacks
   GET    /usage_summary
   DELETE /token_keys/{id}
   DELETE /token_groups/{id}
            │ emits                    │ emits
            ▼                          ▼
   OBSERVATION LAYER: x-ltg-served-model / x-ltg-attempts
                      + GET /spend/events ledger
  • Management plane — api.telnyx.com/v2/llm_token_gateway, authenticated with your Telnyx account API key via the Telnyx Python SDK. This is where group and key lifecycle, budgets, and reporting live.
  • Inference plane — llm.telnyx.com/v1, OpenAI-compatible, authenticated with only a token key. It rejects account keys. Your application code never needs to hold the account credential.
  • Observation layer — headers on the live path, spend events on the ledger. Same story, two sources.

One retry layer, one bill, one story

Here's the detail that saves people real money: the sample builds its OpenAI client with max_retries=0.

If your SDK retries on top of the gateway, those are additional requests stacked on top of gateway attempts. You get double-billed, and your attempt counts are inflated — the gateway's careful accounting now includes retries it never authorized. Because the gateway owns retries and fallbacks, the client should get out of the way. One retry layer, one bill, one story.

# abridged — full runnable code in the repo
llm = OpenAI(
    base_url="https://llm.telnyx.com/v1",
    api_key=token_key,   # token key ONLY — the inference plane rejects account keys
    max_retries=0,       # the gateway owns retries; SDK retries would double-bill
)

The physics of failure: definite vs. uncertain

The gateway doesn't treat all failures the same, and the difference is the whole design:

  • Definite no-work failures — pre-dispatch errors, 429, 529. The request never did any work, so the gateway retries the same model.
  • Uncertain failures — timeout, 408, 500, 502, 503, 504. The request may have landed and been processed, so retrying the same model risks double-processing. The gateway steps to the next fallback instead.
  • Context-length rejection is terminal. The same context will be rejected again, so the gateway does not fall back — it fails fast.

Policy has guardrails, asserted in gateway.py: fallback chain depth ≤ 3, max_attempts between 1 and 5, no self-reference, no duplicates, every fallback model present in allowed_models, and a worst-case attempt ceiling of max_attempts × (1 + longest chain) ≤ 8. Send an invalid config and the gateway answers with a 400 — policy is enforced, not suggested.

Reservations: watch the budget settle

Budget and TPM capacity for planned attempts are reserved up front and released at settlement. After each call, the sample GETs the token group and prints the raw budget/reservation fields, so you can see the reservation appear while attempts are in flight and release when the request settles — regardless of exact field naming on the API.

The forced failure: near-deterministic by construction

You can't demo failover by waiting for a provider outage. The forced-failure run reuses the provisioned group and key and sends a streaming request with the per-request overlay x-ltg-request-timeout: 1. Streaming matters because the deadline applies to the first token — the failure trips fast and repeatably, no chaos engineering required.

The overlay headers (x-ltg-max-attempts, x-ltg-retry-delay, x-ltg-backoff, x-ltg-request-timeout) let you adjust resilience behavior per request without touching group policy.

raw = llm.chat.completions.with_raw_response.create(
    model="kimi-k3",
    messages=messages,
    extra_headers={"x-ltg-request-timeout": "1"},  # force the primary to miss its deadline
)
response = raw.parse()

print(response.model)                          # what the SDK reports
print(raw.headers.get("x-ltg-served-model"))   # what actually served it
print(raw.headers.get("x-ltg-attempts"))       # how many attempts it took

Run it and watch the headers: x-ltg-served-model flips from the primary to the fallback, and x-ltg-attempts climbs from 1 to 2.

Reading the proof

The headers prove the live path. The ledger proves the record:

events = client.llm_token_gateway.spend.events.list(token_group_id=group_id)

for event in events.data:
    print(
        event.requested_model,  # what the client asked for
        "->", event.model,      # what actually served it
        "| attempts:", event.attempts,
        "| status:", event.status,
        "| cache_hit:", event.cache_hit,
    )

requested_model and model on the same row is the audit sentence, written by the system rather than reconstructed by you. GET /usage/summary rolls the same story up into data.totals, data.by_day, and data.by_model when you need the dashboard view. Mutations are concurrency-safe too: the sample fetches the group's ETag and sends it as If-Match (with an Idempotency-Key) on the cache_ttl PATCH and on cleanup deletes.

Setup

  1. Prerequisites: Python 3.10+, a Telnyx account with AI Gateway access, and your Telnyx API key.
  1. Clone and install:
git clone https://github.com/team-telnyx/telnyx-code-examples
cd telnyx-code-examples/gateway-fallbacks
python -m venv .venv && source .venv/bin/activate
pip install telnyx openai
export TELNYX_API_KEY=your_key_here
  1. Provision and assert policy: python gateway.py creates the token group and key, asserts the resilience semantics, and runs the cache_ttl PATCH and the invalid-config 400 proof.
  1. Send traffic and read the evidence: python inference.py runs the baseline call and the forced-failure streaming call, prints the response headers, the ledger rows, and the reservation state, then revokes the key and deletes the group with If-Match ETags.

Conclusion

The pattern here is bigger than fallbacks. Put resilience in policy at the gateway, not in scattered client code. Give it one retry layer so your bill and your attempt counts tell the truth. And demand observability as a first-class property — headers on the live path, a ledger on the record — so that when someone asks "did failover actually happen?", the answer is a fact, not a guess.

That's the promise of AI Communications Infrastructure: the same accountability telecom networks have always had about what happened on the wire, applied to model inference. Clone the sample, trip your own primary, and read the headers for yourself: https://github.com/team-telnyx/telnyx-code-examples/tree/main/gateway-fallbacks

Ready to build with low-latency voice AI?

Join developers building the future of real-time conversations