Voice AI

Build an Omnichannel Context-Carrying AI Agent With Telnyx

Your customer emails about a booking change at 9 AM. At noon, she calls. The agent asks: "How can I help you?" She explains the entire issue again. At 2 PM, she texts for an update. Another agent. Another explanation. By the end of the day, she has repeated herself three times across three channels and is considering switching providers.

This is the default experience at most companies. Every channel is a silo. Every interaction starts from zero.

This example builds the opposite: a single AI agent that carries full context across email, voice, and SMS. When Sarah emails about changing booking BK-2024-1234, then calls an hour later, the AI greets her with: "Hi Sarah! I see you emailed us about changing booking BK-2024-1234 from October 15 to October 22. Are you calling about the same request?"

One Flask app. One API key. Three channels. Zero repeated context.

What This Example Builds

ComponentDetail
LanguagePython (Flask)
ChannelsEmail, Voice, SMS
AITelnyx Inference API (OpenAI-compatible, tool-calling)
Context storeSQLite (cross-channel conversation history)
Identity registrySQLite (email-to-phone customer resolution)
Voice inputgather_using_ai (HD voice + AI transcription)
DashboardReal-time SSE with Customer Journey, Agent Activity, and Channel Timeline panels
Demo modeRuns the full journey without any credentials
Lines of code~1,100 (app.py + demo_server.py + HTML dashboard)

Why This Matters

The business cost of siloed channels is real and measurable:

Customers hate repeating themselves. Harvard Business Review research consistently shows that the single biggest driver of customer disloyalty is making people repeat information across channels. Every time a customer re-explains their issue, their satisfaction drops and their likelihood of churning increases.

Agents waste time on context recovery. When a customer calls after emailing, a human agent spends the first 30-60 seconds asking "Can you tell me what this is about?" and pulling up records. An AI agent without context does the same thing. That is wasted time on every single cross-channel interaction.

The fix is identity resolution plus shared memory. If the system knows that sarah.chen@example.com and +15555550199 are the same person, and if every message from every channel is stored against that person's ID, then any channel can pick up where the last one left off. This is not a novel idea. It is the standard architecture for omnichannel CRM. What is new is that you can build a working version of it in a single Python file using a single API key.

Products Used

  • Telnyx Email API -- receive inbound emails via webhook, send replies programmatically
  • Telnyx SMS API -- receive inbound texts via webhook, send confirmation messages
  • Telnyx Voice API -- receive inbound calls, answer, speak with HD voice, gather speech input
  • Telnyx AI Inference API -- OpenAI-compatible chat completions with tool-calling (send_email, send_sms, make_call, resolve_issue)

All four APIs authenticate with the same TELNYX_API_KEY. One platform, one bill. Telnyx is AI Communications Infrastructure -- voice, messaging, email, and AI inference behind a single authentication surface.

Architecture

                    ┌───────────────────────────────────────────────┐
                    │            Telnyx Platform                    │
                    │  ┌─────────┐ ┌─────────┐ ┌─────────────────┐ │
  Customer ────────►│  │ Email   │ │ Voice   │ │ SMS / Messaging │ │
  (email/call/text) │  │ API     │ │ API     │ │ API             │ │
                    │  └────┬────┘ └────┬────┘ └───────┬─────────┘ │
                    │       │           │              │            │
                    │       │     Webhooks (POST)      │            │
                    └───────┼───────────┼──────────────┼────────────┘
                            │           │              │
                            ▼           ▼              ▼
                    ┌───────────────────────────────────────────────┐
                    │              Flask App (app.py)               │
                    │                                               │
                    │  Layer 1: Customer Identity Registry          │
                    │  ┌─────────────────────────────────────────┐  │
                    │  │ SQLite: customers (id, name, email, phone)│ │
                    │  │ resolve_customer(email=..., phone=...)   │  │
                    │  └─────────────────────────────────────────┘  │
                    │                                               │
                    │  Layer 2: Cross-Channel Context Store         │
                    │  ┌─────────────────────────────────────────┐  │
                    │  │ SQLite: conversations                    │  │
                    │  │ (customer_id, channel, role, content, ts)│  │
                    │  └─────────────────────────────────────────┘  │
                    │                                               │
                    │  Layer 3: Inbound Webhook Handlers            │
                    │  ┌──────────┐ ┌──────────┐ ┌──────────────┐  │
                    │  │/webhooks │ │/webhooks │ │/webhooks     │  │
                    │  │/email    │ │/voice    │ │/messaging    │  │
                    │  └──────────┘ └──────────┘ └──────────────┘  │
                    │                                               │
                    │  Layer 4: AI Brain (Telnyx Inference)         │
                    │  ┌─────────────────────────────────────────┐  │
                    │  │ Tool-calling: send_email, send_sms,     │  │
                    │  │ make_call, resolve_issue                 │  │
                    │  │ Agentic loop until finish_reason: stop   │  │
                    │  └─────────────────────────────────────────┘  │
                    │                                               │
                    │  SSE → Browser Dashboard (3 panels)          │
                    └───────────────────────────────────────────────┘

The four layers work together on every inbound message:

  1. Identity Registry resolves the sender's email or phone to a customer ID.
  2. Context Store retrieves every prior message that customer has sent or received, across all channels.
  3. Webhook Handler packages the inbound message plus the full history into a prompt for the AI.
  4. AI Brain reads the cross-channel history, generates a contextual response, and calls tools to deliver it on the appropriate channel.

The Customer Identity Registry

The identity registry is a SQLite table that maps email addresses and phone numbers to a single customer ID:

CREATE TABLE customers (
    id         TEXT PRIMARY KEY,
    name       TEXT NOT NULL,
    email      TEXT,
    phone      TEXT,
    created_at TEXT NOT NULL
);

CREATE UNIQUE INDEX idx_cust_email ON customers (email) WHERE email IS NOT NULL;
CREATE UNIQUE INDEX idx_cust_phone ON customers (phone) WHERE phone IS NOT NULL;

The resolve_customer() function is the entry point every webhook handler calls first:

def resolve_customer(email=None, phone=None):
    """Resolve a customer by email or phone. Returns customer dict or None."""
    if email:
        customer = lookup_customer_by_email(email)
        if customer:
            return customer
    if phone:
        customer = lookup_customer_by_phone(phone)
        if customer:
            return customer
    return None

When an email arrives from sarah.chen@example.com, the email webhook calls resolve_customer(email="sarah.chen@example.com"). When a call comes in from +15555550199, the voice webhook calls resolve_customer(phone="+15555550199"). Both return the same customer record. That is the entire cross-channel linking mechanism: the customer's email and phone are registered once, and every inbound message on any channel resolves to the same ID.

This design is deliberately simple. In production, you would replace SQLite with your CRM or customer database. The interface stays the same -- given an identifier, return a customer ID.

The Cross-Channel Context Store

Every message from every channel is stored in a single conversations table:

CREATE TABLE conversations (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    customer_id TEXT    NOT NULL,
    channel     TEXT    NOT NULL,    -- 'email', 'sms', 'voice', 'system'
    role        TEXT    NOT NULL,    -- 'user' or 'assistant'
    content     TEXT    NOT NULL,
    timestamp   TEXT    NOT NULL
);

The store_message() function writes every interaction:

def store_message(customer_id, channel, role, content):
    conn = sqlite3.connect(DB_PATH)
    cur = conn.cursor()
    cur.execute(
        "INSERT INTO conversations (customer_id, channel, role, content, timestamp) "
        "VALUES (?, ?, ?, ?, ?)",
        (customer_id, channel, role, content, datetime.now(timezone.utc).isoformat()),
    )
    conn.commit()
    conn.close()

And get_conversation_history() retrieves the complete timeline for a customer, ordered chronologically across all channels:

def get_conversation_history(customer_id):
    conn = sqlite3.connect(DB_PATH)
    conn.row_factory = sqlite3.Row
    cur = conn.cursor()
    cur.execute(
        "SELECT channel, role, content, timestamp FROM conversations "
        "WHERE customer_id = ? ORDER BY timestamp",
        (customer_id,),
    )
    rows = cur.fetchall()
    conn.close()
    return [dict(r) for r in rows]

The key detail: when the AI generates a reply, that reply is also stored with the channel tag. So the history for a given customer might look like:

[email]  user:      [Inbound email] Subject: Change booking BK-2024-1234
[email]  assistant: [Email sent] Subject: Re: Booking Change Request ...
[voice]  assistant: [Contextual greeting] Hi Sarah! I see you emailed us...
[voice]  user:      [Voice] Yes, go ahead and change it to October 22.
[sms]    assistant: [SMS sent] Hi Sarah, your booking has been updated...
[system] assistant: [Issue resolved] Booking changed from Oct 15 to Oct 22.

This timeline is what the AI reads before generating any response. It is the memory that makes context-carrying work.

The Contextual Greeting Generator

This is the core "wow" feature. When a customer calls after emailing (or after any prior channel interaction), the AI generates a greeting that demonstrates awareness of the prior conversation:

def generate_contextual_greeting(customer_id):
    """Generate a contextual greeting that references prior cross-channel
    interactions. This is the core context-carrying feature."""
    history = get_conversation_history(customer_id)
    if not history:
        return "Hello! Thank you for contacting TelnyxDemo Corp. How can I help you today?"

    history_text = "\n".join(
        f"[{msg['channel']}] {msg['role']}: {msg['content']}" for msg in history
    )

    messages = [
        {
            "role": "system",
            "content": (
                "You are an AI customer service agent. A customer is contacting you "
                "on a new channel. Generate a warm, contextual greeting that:\n"
                "1. Addresses them by name if known\n"
                "2. References their specific prior interaction (channel, topic, details)\n"
                "3. Asks if they're contacting about the same issue\n"
                "Keep it to 1-2 sentences. Be specific -- mention booking numbers, dates, "
                "or other details from the history."
            ),
        },
        {
            "role": "user",
            "content": f"Prior conversation history:\n{history_text}\n\n"
                       f"Generate a contextual greeting.",
        },
    ]

    resp = requests.post(
        INFERENCE_URL,
        headers={
            "Authorization": f"Bearer {TELNYX_API_KEY}",
            "Content-Type": "application/json",
        },
        json={"model": AI_MODEL, "messages": messages, "max_tokens": 256},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]

Instead of "How can I help you?" the caller hears: "Hi Sarah! I see you emailed us about changing booking BK-2024-1234 from October 15 to October 22. Are you calling about the same request?"

That greeting is the difference between a siloed IVR and an intelligent agent. And it requires exactly two things: a customer identity that links email to phone, and a message store that spans channels.

The AI Brain -- Tool-Calling

The AI agent uses Telnyx Inference API with OpenAI-compatible tool-calling. Four tools are defined:

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "send_email",
            "description": (
                "Send a detailed email to the customer. Use for formal "
                "acknowledgments, detailed explanations, or when a written "
                "record is needed."
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "subject": {"type": "string", "description": "Email subject line"},
                    "body": {"type": "string", "description": "Email body text"},
                },
                "required": ["subject", "body"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "send_sms",
            "description": (
                "Send a short SMS text message to the customer. Use for quick "
                "status updates, confirmations, or time-sensitive notifications."
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "text": {"type": "string", "description": "SMS message text"},
                },
                "required": ["text"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "make_call",
            "description": (
                "Call the customer and speak a message. Use for complex "
                "resolution, urgent matters, or when a personal touch is needed."
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "speak_text": {"type": "string", "description": "Text to speak when the call is answered"},
                },
                "required": ["speak_text"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "resolve_issue",
            "description": "Mark the customer issue as resolved.",
            "parameters": {
                "type": "object",
                "properties": {
                    "summary": {"type": "string", "description": "Brief summary of how the issue was resolved"},
                },
                "required": ["summary"],
            },
        },
    },
]

The execute_tool() function dispatches each tool call to the corresponding Telnyx API:

def execute_tool(tool_name, tool_input, customer):
    if tool_name == "send_email":
        send_email(customer["email"], tool_input["subject"], tool_input["body"])
        store_message(customer["id"], "email", "assistant",
                      f"[Email sent] Subject: {tool_input['subject']}\n{tool_input['body']}")
        return f"Email sent to {customer['email']} with subject: {tool_input['subject']}"
    elif tool_name == "send_sms":
        send_sms(customer["phone"], tool_input["text"])
        store_message(customer["id"], "sms", "assistant", f"[SMS sent] {tool_input['text']}")
        return f"SMS sent to {customer['phone']}: {tool_input['text']}"
    elif tool_name == "make_call":
        make_call(customer["phone"], tool_input["speak_text"])
        store_message(customer["id"], "voice", "assistant",
                      f"[Call initiated] Will speak: {tool_input['speak_text']}")
        return f"Call initiated to {customer['phone']}"
    elif tool_name == "resolve_issue":
        store_message(customer["id"], "system", "assistant",
                      f"[Issue resolved] {tool_input['summary']}")
        return f"Issue resolved: {tool_input['summary']}"
    return f"Unknown tool: {tool_name}"

Notice that every tool call also writes to the context store. When the AI sends an email reply, that reply is stored as an [email] assistant message. When it sends an SMS confirmation, that is stored as an [sms] assistant message. This means the next time the customer contacts on any channel, the AI can see every action it previously took.

The agentic loop continues calling tools until the model returns finish_reason: stop:

while True:
    data = call_inference(messages)
    choice = data["choices"][0]
    msg = choice["message"]

    if not msg.get("tool_calls"):
        break  # No more tools to call -- done

    messages.append(msg)
    for tc in msg["tool_calls"]:
        fn = tc["function"]
        tool_name = fn["name"]
        tool_input = json.loads(fn.get("arguments", "{}"))
        result = execute_tool(tool_name, tool_input, customer)
        messages.append({"role": "tool", "tool_call_id": tc["id"], "content": result})

gather_using_ai -- HD Voice + AI Transcription

When a call is answered, the app uses gather_using_ai to simultaneously speak the contextual greeting and capture the caller's response:

requests.post(
    f"https://api.telnyx.com/v2/calls/{call_control_id}/actions/gather_using_ai",
    headers={
        "Authorization": f"Bearer {TELNYX_API_KEY}",
        "Content-Type": "application/json",
    },
    json={
        "greeting": greeting,
        "voice": "Telnyx.Ultra.00a77add-48d5-4ef6-8157-71e5437b282d",
        "language": "en",
        "parameters": {
            "type": "object",
            "properties": {
                "customer_response": {
                    "type": "string",
                    "description": "What the customer said in response",
                },
            },
            "required": ["customer_response"],
        },
        "assistant": {
            "instructions": (
                "You are a silent listener. Your ONLY job is to capture "
                "EXACTLY what the customer says. Do NOT ask follow-up "
                "questions. Do NOT respond or have a conversation. Just "
                "capture their response word-for-word and complete."
            ),
        },
    },
    timeout=10,
)

gather_using_ai combines three operations into a single API call:

  1. Speak the greeting with a premium HD voice (Telnyx.Ultra)
  2. Listen to the caller's response using AI-powered speech recognition
  3. Extract structured data from the speech (the parameters schema)

When the caller finishes speaking, Telnyx fires a call.ai_gather.ended webhook with the transcribed text in message_history. The app extracts the speech, stores it in the context store, and feeds it to the AI brain for processing.

This approach is simpler than chaining separate speak + gather commands, and the AI transcription handles natural speech better than DTMF or basic speech recognition.

Try the Demo

The demo runs "Sarah's Booking Change" -- the full email, voice, and SMS journey -- without any Telnyx credentials. Mock APIs simulate every interaction.

Step 1: Clone the repository

git clone https://github.com/team-telnyx/telnyx-code-examples.git
cd telnyx-code-examples/omnichannel-context-carrying-python

Step 2: Install dependencies

pip install -r requirements.txt

Step 3: Start the demo server

python demo/demo_server.py

The demo server starts on port 5555 with mock APIs pre-configured. No .env file needed.

Step 4: Open the dashboard and trigger the journey

Open http://localhost:5555/ in your browser. Click "Run Demo Journey" in the header.

Step 5: Watch the three-panel dashboard

The dashboard updates in real time via Server-Sent Events:

  • Customer Journey (left panel) -- tracks each step: customer registered, email received, email replied, call received, contextual greeting spoken, SMS confirmation sent, issue resolved
  • Agent Activity (center panel) -- shows the AI's reasoning and tool calls as they happen
  • Channel Timeline (right panel) -- displays every message across email, voice, and SMS in chronological order

The scripted scenario plays out over about 15 seconds:

  1. Sarah emails about changing booking BK-2024-1234 from October 15 to October 22
  2. AI acknowledges via email with a reference number (CR-2024-0891)
  3. Sarah calls from her registered phone number
  4. AI greets her with context: "Hi Sarah! I see you emailed us about changing booking BK-2024-1234..."
  5. Sarah confirms -- "Yes, go ahead and change it to October 22"
  6. AI processes the change, sends an SMS confirmation, and resolves the issue

At no point does Sarah repeat herself.

Run It With Real Credentials

To run against real Telnyx APIs:

cp .env.example .env

Edit .env:

TELNYX_API_KEY=KEY_your_api_key_here
TELNYX_FROM_NUMBER=+15551234567
TELNYX_EMAIL_FROM=agent@yourdomain.com
MESSAGING_PROFILE_ID=your_messaging_profile_id_here
CONNECTION_ID=your_connection_id_here
AI_MODEL=meta-llama/Llama-3.3-70B-Instruct

Configure Telnyx resources:

  1. Phone number -- buy a number at portal.telnyx.com with voice and messaging enabled
  2. Messaging Profile -- create one and assign the number
  3. Call Control Application -- create one with webhook URL pointing to https://your-domain.com/webhooks/voice
  4. Email domain -- verify a sending domain for the Email API
  5. Webhook URLs -- set up email webhook to https://your-domain.com/webhooks/email and messaging webhook to https://your-domain.com/webhooks/messaging

Register a customer:

curl -X POST http://localhost:5000/customers/register \
  -H "Content-Type: application/json" \
  -d '{"id": "cust_001", "name": "Sarah Chen", "email": "sarah@example.com", "phone": "+15551234567"}'

Now when Sarah emails your verified address, the AI replies. When she calls your Telnyx number, the AI greets her with context from the email. When it sends a confirmation, she gets an SMS.

Going to Production

This example is a working prototype. Five areas to address before production:

  1. Webhook signature verification. The production app should verify the Telnyx Ed25519 signature on every inbound webhook using client.webhooks.unwrap(). The example omits this for clarity, but it is mandatory in production to prevent spoofed webhook events.
  1. Task queue for AI processing. The example processes AI inference in background threads. At scale, use Celery or a similar task queue to handle concurrent requests, retries, and backpressure.
  1. CRM integration for identity. Replace the SQLite customer registry with your CRM (Salesforce, HubSpot, a custom database). The resolve_customer() interface stays the same -- given an email or phone, return a customer ID.
  1. Multi-turn voice conversations. The example uses a single gather_using_ai round. For a full conversational voice agent, loop the gather-process-speak cycle or use Telnyx Conversation Relay for continuous streaming.
  1. Idempotency for tool calls. If the AI calls send_sms and the network fails before the tool result is stored, a retry could send a duplicate SMS. Add idempotency keys to outbound API calls and deduplication logic to the tool executor.

Frequently Asked Questions

What is omnichannel context-carrying? It means that when a customer switches from one communication channel to another (email to phone, phone to SMS), the AI agent retains full awareness of prior interactions. The customer never has to repeat information.

How does the AI know that an email sender and a phone caller are the same person? The Customer Identity Registry maps email addresses and phone numbers to a single customer ID. When an email arrives, the system looks up the sender's email. When a call comes in, it looks up the caller's phone number. If both are registered to the same customer, the AI loads their shared history.

What happens if a customer contacts from an unregistered email or phone? The webhook handler returns a 200 response with {"status": "unknown_customer"} and does not process the message further. In production, you could auto-create customer records or route to a human agent for identity verification.

Can I use a different AI model? Yes. Set the AI_MODEL environment variable to any model available on the Telnyx Inference API. The default is meta-llama/Llama-3.3-70B-Instruct. Any model that supports tool-calling with the OpenAI-compatible format will work.

How is the voice greeting generated? When a call is answered, the app calls generate_contextual_greeting(), which retrieves the customer's full cross-channel history and asks the AI to generate a 1-2 sentence greeting that references specific details (booking numbers, dates, prior channel). This greeting is then spoken to the caller using gather_using_ai with a premium HD voice.

Does the demo require Telnyx credentials? No. The demo server (demo/demo_server.py) patches all Telnyx API calls with mock functions that return scripted responses. It runs the full email-voice-SMS journey locally with no external API calls.

How does the dashboard work? The Flask app publishes Server-Sent Events (SSE) on the /stream endpoint for every significant event: inbound messages, identity resolution, AI reasoning, tool calls, and journey milestones. The browser dashboard connects to this SSE stream and renders events into three panels in real time.

Can I add more channels? Yes. The architecture is channel-agnostic. To add WhatsApp, for example, you would add a /webhooks/whatsapp handler that resolves the customer by phone number (same as SMS), stores messages with channel="whatsapp", and adds a send_whatsapp tool to the TOOLS array. The context store and identity registry work without changes.

Ready to build with low-latency voice AI?

Join developers building the future of real-time conversations