Translation apps interpret between two people speaking different languages. Code-switching is different: one caller switches languages mid-conversation and the agent follows them. No interpreter, no restart, no separate STT and TTS vendors — one Telnyx AI Assistant that detects the spoken language on every turn and replies in kind.
The canonical code example is in the Telnyx code examples repo:
https://github.com/team-telnyx/telnyx-code-examples/tree/main/ai-multilingual-code-switching-agent-python
The Use Case: One Caller, Multiple Languages
A customer calls a support line. They start in English, switch to Spanish for a sensitive question, then switch back. Most voice agents handle one language per call. This one handles all five — English, Spanish, Portuguese, Hindi, and Mandarin — in a single conversation, switching on every turn.
The agent detects the caller's language from their speech, replies in that language, and switches the moment the caller switches. No language picker menu, no "press 1 for English, 2 for Spanish." The caller just speaks, and the agent follows.
How It Works
The entire code-switching behavior lives in the assistant's instructions — there is no application-layer STT/LLM/TTS pipeline. The Flask app provisions the assistant and serves a simple page showing the phone number. The conversation runs entirely on Telnyx.
Inbound phone call ──► Telnyx Voice AI Assistant
│
▼
┌────────────────────────┐
│ STT (deepgram/nova-3) │ → auto-detects spoken language
│ LLM (instructions) │ → replies in same language,
│ │ switches on caller's switch
│ TTS (voice ultra katie) │ → speaks reply in detected language
└────────┬───────────────┘
│
▼
Caller hears reply
in their languageThe Assistant Instructions
The code-switching logic is plain English in the instructions field:
you are a friendly multilingual voice agent for a global customer support line.
you can speak english, spanish, portuguese, hindi, and mandarin.
listen carefully to the caller. detect the language they are speaking on every turn.
reply in the same language the caller is using right now.
if the caller switches language mid-conversation or mid-sentence, switch with them.
if you are unsure which language to use, ask in english "which language would you prefer".
keep replies short and natural. this is a phone call.
do not translate unless the caller asks. you are not an interpreter — you are the agent.
if a caller mixes two languages in one sentence, reply in the dominant language.
never say the name of the language out loud unless asked.The LLM follows these instructions on every turn. The STT transcribes in whatever language the caller speaks. The TTS renders the reply in that language. No routing, no language detection code in the app.
Key Technical Decisions
Why deepgram/nova-3 with language: "auto"
Per the AI Assistants docs: "To enable a multilingual agent, set the transcription model to deepgram/nova-3." The language: "auto" setting lets nova-3 detect the spoken language on every turn. Note: language: "multi" is not valid — the API returns 10015 Bad Request. Use "auto".
Why voice ultra katie in the instructions
The existing phone assistant examples in the repo (ai-lab-results-notification-voice-python, ai-pci-protected-payment-collection-python) use voice ultra katie in the instructions. Ultra supports 36+ languages, so one voice handles all five demo languages.
Why moonshotai/Kimi-K2.6
A native Telnyx model — no external API key required. Available out of the box.
Architecture
The Flask app is minimal. Since the Voice AI Assistant handles the entire conversation (STT, LLM, TTS, telephony), the app only needs to:
- Create/reuse the assistant (
provision_assistant.py) - Serve a page showing the inbound phone number (
GET /) - Log webhook events (
/webhooks/call)
There is no call-control logic, no application-layer STT, no manual TTS. The conversation runs entirely on Telnyx.
Run It
git clone https://github.com/team-telnyx/telnyx-code-examples.git
cd telnyx-code-examples/ai-multilingual-code-switching-agent-python
cp .env.example .env # fill in TELNYX_API_KEY, phone number, connection ID
python provision_assistant.py # creates the assistant
python app.py # starts on http://127.0.0.1:5050Open the browser, see the phone number, call it. Speak in English, switch to Spanish, switch back.
Related Examples
ai-real-time-translation-bridge-python— two-caller interpreter (different problem)ai-language-learning-phone-tutor-python— phone-based language tutorlanguage-learning-flashcards-python— browser-based pronunciation scoring