Building a Natural Language IVR with Telnyx Call Control and AI Inference

Traditional IVR systems are universally disliked. Callers are forced to listen to long menus and press numbers or say rigid keywords: "Press 1 for billing, Press 2 for support." If they miss an option, they start over. It is a friction-heavy experience that everyone tolerates and nobody enjoys.

With Telnyx's AI Communications Infrastructure, you can replace that paradigm with a natural language conversation. Instead of navigating a phone tree, callers simply say what they need — "I have a question about my invoice" — and the system routes them to the right department. In this post, we'll walk through the Voice IVR with Agent Backend, a sample app built with Python, Flask, Call Control, and AI Inference.

What the App Does

The Voice IVR with Agent Backend is a Flask application that handles inbound calls with a conversational IVR. When a caller dials your Telnyx number, the application:

1. Answers the call using Call Control.
2. Looks up the menu configuration for the dialed number in a Key-Value (KV) store.
3. Uses an LLM to generate a dynamic, conversational greeting tailored to the business.
4. Uses Call Control's gather_using_speech to listen to the caller's request.
5. Uses an LLM to route the caller's intent to the correct department.
6. Transfers the call using Call Control.

If the LLM cannot understand the caller after a configurable number of retries, the app falls back to a keyword-matching algorithm and ultimately transfers to a default operator number.

How It Works

The application is built around an IVRAgent class that manages the state of each active call. While implemented as a standard Python class for portability, it mirrors the pattern of the Telnyx Agent SDK.

1. KV-Backed Menu Configuration

Instead of hardcoding menu options, the app uses an in-memory Key-Value store (MENU_CONFIG_KV) to map dialed phone numbers to business-specific configurations. Each config includes the business name, a fallback greeting, and a list of departments with descriptions, transfer numbers, and keywords.

MENU_CONFIG_KV: dict[str, dict] = {
    "+180****0000": {
        "business_name": "Acme Corp",
        "greeting": "Welcome to Acme Corp. How can I help you today?",
        "departments": [
            {
                "name": "billing",
                "description": "questions about invoices, payments, or account charges",
                "transfer_to": "+180****1000",
                "keywords": ["billing", "invoice", "payment", "charge", "bill", "account"],
            },
            # ...support, sales...
        ],
    },
}

In production, you would swap this dictionary for Redis, a database, or Telnyx KV. The management API exposes PUT /api/menu-config/<phone_number> to update configurations dynamically.

2. LLM-Powered Dynamic Greeting

When the call connects, the agent generates a conversational greeting using Telnyx AI Inference. The generate_dynamic_menu_prompt function builds a system prompt from the KV config and asks the LLM to produce a brief, natural greeting.

def generate_dynamic_menu_prompt(menu_config: dict) -> str:
    departments = menu_config.get("departments", [])
    dept_list = "\n".join(
        f"- {d['name']}: {d['description']}" for d in departments
    )
    return (
        f"You are an IVR assistant for {menu_config.get('business_name', 'the company')}. "
        f"Available departments:\n{dept_list}\n\n"
        f"Greet the caller briefly and ask how you can help. "
        f"Keep it conversational and under 2 sentences."
    )

The LLM generates the greeting via the OpenAI-compatible Telnyx Inference binding:

completion = telnyx.ai.openai.chat.completions.create(
    model="telnyx-llm",
    messages=[{"role": "user", "content": prompt}],
    max_tokens=100,
    temperature=0.7,
)

If the LLM fails, the app gracefully falls back to the static greeting from the KV config.

3. Speech Gather with Call Control

After speaking the greeting, the app uses Call Control's gather_using_speech to listen to the caller's response. This primitive plays a prompt and captures the caller's speech, transcribing it for the LLM.

def gather_speech(call_control_id: str, prompt: str) -> None:
    telnyx.Call.gather_using_speech(
        call_control_id,
        payload=prompt,
        voice="female-en-US",
        language="en-US",
        max_duration=15,
    )

4. LLM-Powered Intent Routing

When the gather completes, the transcribed speech is passed to route_intent_with_llm, which asks the LLM to match the caller's intent to a department. The LLM is instructed to respond with only the department name for reliable parsing.

def route_intent_with_llm(user_input: str, menu_config: dict) -> dict | None:
    system_prompt = (
        "You are an intent router for an IVR system. "
        "Given the caller's input, determine which department they need. "
        "Respond with ONLY the department name (lowercase) or 'unknown'.\n\n"
        f"Departments:\n{dept_descriptions}"
    )
    completion = telnyx.ai.openai.chat.completions.create(
        model="telnyx-llm",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_input},
        ],
        max_tokens=20,
        temperature=0.1,
    )
    intent = completion.choices[0].message.content.strip().lower()
    for dept in departments:
        if dept["name"] in intent:
            return dept

If the LLM fails or returns "unknown," the function falls back to keyword matching using the keywords defined in the KV config. If no match is found after max_turns (default 3), the call is transferred to a default operator number.

5. Webhook Verification

Security is critical when handling inbound webhooks. The verify_telnyx_webhook decorator validates the Telnyx Ed25519 signature on every request to the /webhooks/voice endpoint.

def verify_telnyx_webhook(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        signature = request.headers.get("Telnyx-Signature-Ed25519")
        timestamp = request.headers.get("Telnyx-Signature-Timestamp")
        telnyx.Webhook.construct_event(
            payload, signature, timestamp, TELNYX_PUBLIC_KEY
        )
        return f(*args, **kwargs)
    return decorated

6. The IVR Agent State Machine

The IVRAgent class tracks the state of each active call:

• on_connect(): Triggered when the call is answered. Fetches the menu config, generates the LLM greeting, speaks to the caller, and initiates the speech gather.
• on_gather_ended(speech): Triggered when the gather completes. Routes intent via LLM, transfers if matched, retries if not, and falls back to a default transfer after max_turns.

Setup

To run this locally, you need a Telnyx account, a Call Control Application, a purchased phone number, and an API key.

1. Clone the repository:

``bash git clone https://github.com/team-telnyx/telnyx-code-examples.git cd telnyx-code-examples/voice-ivr-with-agent-backend ``

1. Set up your environment:

``bash python3 -m venv venv source venv/bin/activate pip install -r requirements.txt cp .env.example .env ``

1. Configure your .env file:

Fill in your Telnyx API Key, Public Key, Connection ID, and default transfer number.

1. Run the application:

``bash python app.py ``

1. Expose with ngrok:

``bash ngrok http 5000 ``

Point your Telnyx Call Control Application webhook to https://<your-ngrok-url>.ngrok-free.app/webhooks/voice.

Conclusion

Traditional IVR systems force callers into rigid phone trees. The Voice IVR with Agent Backend demonstrates how Telnyx's AI Communications Infrastructure lets you replace that with a natural language conversation — combining Call Control for voice orchestration, AI Inference for dynamic greetings and intent routing, and an Agent SDK-style state machine for per-call lifecycle management, all in a single Python application.

Check out the full code on GitHub and start building your own conversational IVR today.