Build an AI Language Learning Phone Tutor with Telnyx Voice AI and AI Inference

Language learning apps work fine on a screen, but the hardest part of picking up a new language is the thing screens avoid: actually speaking out loud, in real time, to someone who responds. This walkthrough builds a Flask app that turns a phone call into a language practice session — the caller dials a number, picks a language, and talks to an AI tutor that responds in the target language with English support, adjusts difficulty as the conversation grows, and corrects mistakes gently. The whole loop runs on Telnyx Call Control for the voice path and AI Inference for the conversation, with text-to-speech rendering the tutor's responses back to the caller.

The canonical code example lives in the Telnyx code examples repo:

https://github.com/team-telnyx/telnyx-code-examples/tree/main/ai-language-learning-phone-tutor-python

What This Example Builds

A single-file Flask app with three endpoints, driven by a webhook state machine:

The call flow runs in four stages. An inbound call arrives and is answered. Text-to-speech greets the caller and asks them to pick a language by pressing a digit. The caller speaks (or dials a number), and their speech is transcribed via Call Control's gather action. The transcript goes to AI Inference with a system prompt tuned for language tutoring — the model responds in the target language with English support, corrects mistakes, and gradually raises difficulty. The response is spoken back to the caller via TTS, and the loop repeats until the caller hangs up.

Why This Matters

The phone tutor exercises two Telnyx AI products in a real-time, full-duplex loop — Voice AI for the call path and AI Inference for the conversation — without any third-party telephony or LLM plumbing. That makes it a useful reference for anyone building conversational voice applications where the AI is not just reacting to a single prompt but holding a multi-turn conversation with state.

Developers use this pattern as the foundation for:

• Language practice hotlines — Give learners a number they can call anytime for low-stakes speaking practice
• Customer support voice agents — Replace rigid IVR menus with an AI that asks clarifying questions and responds naturally
• Interactive voice surveys — Ask open-ended questions and let the AI probe for richer answers than DTMF menus allow
• Accessibility services — Offer spoken-language interfaces for users who can't or won't use a screen
• Educational tutoring — Extend the tutor prompt to any subject and let callers practice on any phone

Products Used

telnyx_products: [Voice AI, AI Inference]

Telnyx Call Control drives the call — answer, gather speech or DTMF, play TTS, and hangup — all through a webhook-driven state machine so the app never holds a socket open for the call's duration. Telnyx AI Inference exposes an OpenAI-compatible chat completions endpoint, so the conversation context is just a list of {role, content} messages sent to POST /v2/ai/chat/completions.

Architecture

  Inbound Phone Call
        │
        ▼
  ┌──────────────────┐
  │ call.initiated    │  app answers the call
  └────────┬─────────┘
           │
           ▼
  ┌──────────────────┐
  │ call.answered     │  TTS greeting: "Press 1 for Spanish, 2 for French…"
  └────────┬─────────┘
           │
           ▼
  ┌──────────────────┐
  │ call.speak.ended  │  gather DTMF (1 digit) or speech
  └────────┬─────────┘
           │
           ▼
  ┌──────────────────┐
  │ call.gather.ended│  parse selection → seed system prompt
  │                  │  → AI Inference → TTS response
  └────────┬─────────┘
           │
           ▼
  ┌──────────────────┐
  │ call.speak.ended │  gather speech (caller's turn)
  │                  │  ◄── conversation loop
  └────────┬─────────┘
           │
           ▼
       call.hangup

The Code

The full app is 108 lines in a single app.py. The key pieces:

Configuration and call state. The Telnyx API key and public key are loaded from the environment; the public key is used to verify incoming webhook signatures. Two in-memory stores track active calls and completed sessions, with a background TTL thread that prunes stale entries after an hour so memory does not grow without bound:

client = telnyx.Telnyx(api_key=os.getenv("TELNYX_API_KEY"), public_key=os.getenv("TELNYX_PUBLIC_KEY"))
active_calls = {}
session_history = []

def _start_ttl_cleanup(*stores, ttl_seconds=3600, interval=300):
    def _cleanup():
        while True:
            _ttl_time.sleep(interval)
            cutoff = _ttl_time.time() - ttl_seconds
            for store in stores:
                expired = [k for k, v in store.items()
                           if isinstance(v, dict) and v.get("_ts", _ttl_time.time()) < cutoff]
                for k in expired:
                    store.pop(k, None)
    threading.Thread(target=_cleanup, daemon=True).start()

Webhook signature verification. Every incoming webhook is verified against the Telnyx Ed25519 signature before the app trusts the event. Unverified requests get a 401 and the handler returns immediately:

@app.route("/webhooks/voice", methods=["POST"])
def handle_voice():
    try:
        client.webhooks.unwrap(request.get_data(as_text=True), headers=dict(request.headers))
    except Exception:
        return jsonify({"error": "invalid signature"}), 401
    payload = request.get_json()
    ...

AI Inference call. A thin wrapper over the OpenAI-compatible chat completions endpoint. The conversation is passed as a list of messages — the first being the system prompt that defines the tutor's behavior, followed by alternating user/assistant turns. The default max_tokens is 200, which is enough for short phone responses on a non-reasoning model:

INFERENCE_URL = "https://api.telnyx.com/v2/ai/chat/completions"

def call_inference(messages, max_tokens=200):
    resp = requests.post(INFERENCE_URL, headers={
        "Authorization": f"Bearer {TELNYX_API_KEY}",
        "Content-Type": "application/json",
    }, json={
        "model": AI_MODEL, "messages": messages,
        "max_tokens": max_tokens, "temperature": 0.7,
    }, timeout=15)
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]

The call flow state machine. Each Telnyx event triggers the next action. The caller picks a language on the first gather, which seeds the system prompt with the target language and tutor behavior. From then on, every gather's transcribed speech becomes the next user message, the model's reply becomes the next TTS payload, and the loop continues until hangup:

if event_type == "call.initiated" and p.get("direction") == "incoming":
    active_calls[ccid] = {"caller": p.get("from"), "state": "language_select", "conversation": []}
    client.calls.actions.answer(ccid)
elif event_type == "call.answered":
    client.calls.actions.speak(ccid, payload="Welcome to Language Tutor! Press 1 for Spanish, 2 for French, 3 for Japanese, 4 for Mandarin.", voice="female", language_code="en-US")
elif event_type == "call.speak.ended" and call:
    if call["state"] == "language_select":
        client.calls.actions.gather(ccid, input_type="dtmf speech", timeout_secs=10, min_digits=1, max_digits=1)
    else:
        client.calls.actions.gather(ccid, input_type="speech", end_silence_timeout_secs=3, timeout_secs=20, language_code="en-US")
elif event_type == "call.gather.ended" and call:
    if call["state"] == "language_select":
        lang = LANGUAGES.get(digits or speech[:1], LANGUAGES["1"])
        call["language"] = lang
        call["state"] = "tutoring"
        call["conversation"] = [{"role": "system", "content": f"You are a {lang['name']} language tutor. Start with a simple greeting in {lang['name']}, then English translation. Gradually increase difficulty. Correct mistakes gently. Mix {lang['name']} and English. Keep each response short for phone conversation."}]
        intro = call_inference(call["conversation"] + [{"role": "user", "content": "Start the lesson."}])
        call["conversation"].append({"role": "assistant", "content": intro})
        client.calls.actions.speak(ccid, payload=intro, voice="female", language_code="en-US")
    elif call["state"] == "tutoring" and speech:
        call["conversation"].append({"role": "user", "content": speech})
        response = call_inference(call["conversation"])
        call["conversation"].append({"role": "assistant", "content": response})
        client.calls.actions.speak(ccid, payload=response, voice="female", language_code="en-US")
elif event_type == "call.hangup":
    call = active_calls.pop(ccid, None)
    if call and call.get("conversation"):
        session_history.append({"caller": call["caller"], "language": call.get("language", {}).get("name"), "exchanges": len(call["conversation"]) // 2})

The system prompt is the only thing that needs to change to repurpose this app for a different domain — swap "language tutor" for "support agent" or "intake assistant" and the same state machine drives a completely different conversation.

Run It

git clone https://github.com/team-telnyx/telnyx-code-examples.git
cd telnyx-code-examples/ai-language-learning-phone-tutor-python
cp .env.example .env  # add TELNYX_API_KEY, TELNYX_PUBLIC_KEY, TUTOR_NUMBER
pip install -r requirements.txt
python app.py

The server starts on http://localhost:5000. Expose it for webhooks with a tunnel:

ngrok http 5000

Copy the HTTPS URL and configure it in the Telnx Portal:

• Call Control Application → Webhook URL → https://<id>.ngrok.io/webhooks/voice

Then call your Telnyx number from any phone. The app answers, greets you, and asks you to pick a language. Speak or dial a digit, and the tutor starts the lesson.

Check the health and recent sessions:

curl http://localhost:5000/health
# {"status":"ok","active":1,"sessions":0}

curl http://localhost:5000/sessions
# {"sessions":[{"caller":"+12125551234","language":"Spanish","exchanges":6}]}

Choosing a Model

The sample defaults to AI_MODEL=moonshotai/Kimi-K2.6 with max_tokens=200. Kimi-K2.6 is a reasoning model — it spends tokens on internal chain-of-thought before producing a final answer. On a phone call where every second of silence counts, that reasoning budget can consume the entire token window and leave the caller with no spoken response.

For a phone tutor, two practical configurations:

• Keep Kimi-K2.6, raise max_tokens — Set max_tokens=1000 or higher so the model has room for both reasoning and a usable response. Latency goes up, but the tutor still speaks.
• Switch to a non-reasoning model — Set AI_MODEL=meta-llama/Llama-3.3-70B-Instruct (or any other chat model on Telnyx AI Inference) for snappier responses with no reasoning overhead. With max_tokens=200, the caller hears a reply in well under a second of generation.

Set the model and token ceiling in .env:

AI_MODEL=meta-llama/Llama-3.3-70B-Instruct
# Optionally override max_tokens in app.py's call_inference() default

See the available models list for the full catalog.

Available Languages

The app ships with four languages out of the box:

Add more by extending the LANGUAGES dict in app.py and updating the greeting prompt:

LANGUAGES = {
    "1": {"name": "Spanish", "code": "es"},
    "2": {"name": "French", "code": "fr"},
    "3": {"name": "Japanese", "code": "ja"},
    "4": {"name": "Mandarin", "code": "zh"},
    "5": {"name": "German", "code": "de"},
    "6": {"name": "Korean", "code": "ko"},
}

The system prompt is generated from the language name, so any language with a working TTS voice on Telnyx works without further changes.

Going to Production

This example uses in-memory dicts for call state and session history, which is fine for local testing but loses state on restart. For production:

• Persistent state — Replace active_calls and session_history with Redis or PostgreSQL so call state survives restarts and scales across multiple Flask workers
• Model selection — Pick a non-reasoning chat model for sub-second phone responses, or tune max_tokens carefully if you keep a reasoning model. See the models list for latency and context tradeoffs
• TTS language matching — The sample speaks the tutor's responses with language_code="en-US". For authentic pronunciation in the target language, map the language_code to the selected language (e.g. es-ES for Spanish, ja-JP for Japanese) and pick an appropriate voice
• Conversation length limits — Cap call["conversation"] to the last N turns before sending to inference, so long calls do not blow past the model's context window
• Authentication — Add API key validation on /sessions and /health if the app is publicly reachable
• Webhook signature verification — Already implemented via client.webhooks.unwrap; ensure TELNYX_PUBLIC_KEY is set in .env or the app will reject every event
• Error recovery — If call_inference raises (timeout, non-200), speak a graceful "let me try that again" message and re-gather rather than dropping the call
• Monitoring — Add structured logging and alert on inference failures, gather timeouts, and call hangup rates so you catch upstream issues early
• Rate limiting — Protect /webhooks/voice from replay attacks; the signature check is the primary defense, but a per-call_control_id replay guard adds defense in depth