Voice AI

Build an AI Grocery Delivery Assistant on WhatsApp with Telnyx

Grocery delivery apps have a retention problem. Users download the app, order once, forget their password, and go back to the shop. The interface that every phone already has — a messaging app — never gets used for commerce because wiring up a conversational AI agent to a messaging channel is traditionally a stack of three or four vendors: a messaging gateway, an LLM provider, a transcription service, and a phone validation API.

This walkthrough builds FreshCart, a WhatsApp-based grocery delivery assistant powered by an AI agent named Mia. A customer messages a WhatsApp number, browses products, builds a cart, places an order, tracks delivery, handles substitutions, and reorders previous deliveries — all inside a single chat thread. Voice notes work too: say "add bananas, eggs, and sourdough" and Mia transcribes, parses, and adds each item to the cart.

The whole system runs on four Telnyx products through a single unified API — no third-party LLM, no external transcription service, no separate phone validation vendor.

The canonical code example lives in the Telnyx code examples repo:

https://github.com/team-telnyx/telnyx-code-examples/tree/main/whatsapp-grocery-delivery-nodejs

What This Example Builds

A Node.js Express server with a WhatsApp webhook handler, an agentic AI tool-calling loop, and a 3-panel real-time dashboard:

ComponentPurpose
WhatsApp webhookReceives inbound messages (text, interactive replies, voice notes)
AI agent (Mia)12-tool agentic loop — browse, cart, order, track, substitute, reorder
DashboardProduct catalog, WhatsApp conversation mirror, delivery pipeline
SSE streamReal-time event push to the browser dashboard

The AI agent uses tool calling — Mia receives the customer's message, decides which tool to invoke (browse products, add to cart, place order, etc.), executes it, and responds via WhatsApp with text, interactive list messages, or quick-reply buttons depending on context.

Why This Matters

WhatsApp has 2 billion monthly active users. For many markets — grocery, pharmacy, food delivery — WhatsApp is where customers already are. Building a full commerce experience inside WhatsApp means no app download, no account creation, no forgotten passwords. The customer messages a number and starts shopping.

The technical pattern here — an agentic tool-calling loop over a messaging channel — is the foundation for any conversational commerce application:

  • Grocery and food delivery — Browse, order, track, substitute, reorder
  • Pharmacy — Prescription refills, drug interaction checks, delivery scheduling
  • Restaurant ordering — Menu browsing, customization, delivery tracking
  • Retail — Product search, size/color selection, order tracking, returns
  • Service booking — Appointment scheduling, rescheduling, cancellations

The system prompt and tool definitions are the only things that change. The agentic loop, the WhatsApp integration, and the dashboard pattern stay the same.

Products Used

telnyx_products: [WhatsApp Business API, AI Inference, Number Lookup, Audio Transcription]

WhatsApp Business API — Sends and receives messages via POST /v2/messages/whatsapp. Supports text, interactive list messages (for product browsing), and quick-reply buttons (for substitution decisions and delivery confirmations).

AI Inference — Powers the agentic tool-calling loop via POST /v2/ai/chat/completions with OpenAI-compatible function calling. The model (GLM-5.3-Flash) receives the conversation history and tool definitions, decides which tool to call, and the server executes it. The loop runs until the model responds with a final text message.

Number Lookup — Validates and normalizes customer phone numbers at registration via GET /v2/number_lookup/{phone}. Returns E.164 format so the app never stores mismatched local vs international phone formats.

Audio Transcription — Transcribes WhatsApp voice notes via POST /v2/ai/audio/transcriptions using the Whisper model. The customer says "add bananas, eggs, and sourdough" and the transcription feeds into the AI agent as text, which parses the items and calls add_to_cart for each one.

Architecture

  WhatsApp Message (text / voice note / button tap)
        |
        v
  +-----------------------+
  | /webhooks/whatsapp    |  Express webhook handler
  | (inbound message)     |  - dedup, extract text, transcribe audio
  +----------+------------+
             |
             v
  +-----------------------+
  | AI Inference           |  Agentic tool-calling loop
  | (GLM-5.3-Flash)       |  - 12 tools available
  +----------+------------+  - runs until final text response
             |
     +-------+-------+
     |               |
     v               v
  [Tool execution]  [WhatsApp reply]
  - browse_products  - sendText()
  - add_to_cart      - sendList()
  - place_order      - sendButtons()
  - track_delivery
  - etc.
             |
             v
  +-----------------------+
  | SSE -> Dashboard      |  Real-time 3-panel view
  | (browser)             |  - catalog, chat, pipeline
  +-----------------------+

The AI Tools

Mia has 12 tools at her disposal. The AI model decides which tool to call based on the customer's message:

ToolPurposeWhatsApp Output
browse_productsBrowse by category or search by nameInteractive list message
add_to_cartAdd items (supports bulk from voice notes)Text confirmation
view_cartShow cart contents and running totalFormatted text
update_cartChange quantity or remove itemsText confirmation
get_delivery_slotsAvailable delivery windowsInteractive list message
place_orderCheckout from cart with selected slotText with order number
track_deliveryDelivery status and ETAText with status timeline
reschedule_deliveryChange delivery windowQuick-reply buttons
handle_substitutionAccept/reject item replacementsQuick-reply buttons (3 options)
reorderReorder items from a past deliveryQuick-reply buttons to confirm
register_customerRegister new customer + Number Lookup validationText
send_whatsappSend formatted message (text, list, or buttons)Varies

The Tool-Calling Loop

The core of the AI agent is a loop that runs until the model returns a final text response instead of a tool call:

async function processInbound(db, ssePublish, fromNumber, text) {
  const messages = await getConversationHistory(db, fromNumber);
  messages.push({ role: "user", content: text });

  for (let turn = 0; turn < MAX_TURNS; turn++) {
    const response = await chatCompletion(messages);
    const choice = response.choices[0];

    if (choice.finish_reason === "tool_calls") {
      for (const toolCall of choice.message.tool_calls) {
        const result = await executeTool(db, ssePublish, fromNumber, toolCall);
        messages.push({ role: "tool", content: JSON.stringify(result), tool_call_id: toolCall.id });
      }
    } else {
      // Final text response — send to WhatsApp
      const reply = choice.message.content;
      await sendText(fromNumber, reply);
      await storeMessage(db, fromNumber, "assistant", reply);
      break;
    }
  }
}

Each tool call is executed server-side — database queries for products, cart mutations, order placement — and the result is fed back to the model so it can decide what to do next. A single customer message might trigger multiple tool calls: "add eggs and milk" causes the model to call add_to_cart twice, then respond with a confirmation.

Voice Note Ordering

When a customer sends a WhatsApp voice note, the webhook detects msgType === "audio" and routes the audio URL to Telnyx Audio Transcription:

const resp = await fetch("https://api.telnyx.com/v2/ai/audio/transcriptions", {
  method: "POST",
  headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
  body: JSON.stringify({ url: audioUrl, language: "en", model: "whisper" }),
});
const data = await resp.json();
return data.text;

The transcription feeds into the AI agent as text with a [Voice note transcription]: prefix. The agent parses the items naturally — "add bananas, eggs, and sourdough" becomes three separate add_to_cart calls.

Phone Number Validation

New customers register through the dashboard with their phone number. Before storing it, the server calls Telnyx Number Lookup to validate and normalize:

const nlResp = await fetch(
  `https://api.telnyx.com/v2/number_lookup/${encodeURIComponent(phone)}`,
  { headers: { Authorization: `Bearer ${apiKey}` } }
);
const nlData = await nlResp.json();
if (nlData.data.valid_number === false) {
  return res.status(422).json({ error: "Invalid phone number" });
}
const e164Phone = nlData.data.phone_number; // normalized E.164

This prevents the common bug where a customer registers with a local format (0626781895) but WhatsApp sends messages from the international format (+31626781895), creating two separate customer records that never match.

Run It

git clone https://github.com/team-telnyx/telnyx-code-examples.git
cd telnyx-code-examples/whatsapp-grocery-delivery-nodejs
cp .env.example .env  # add your keys
npm install
node server.js

The server starts on http://localhost:3000. Expose it for webhooks:

ngrok http 3000

Copy the HTTPS URL and configure it in the Telnyx Portal:

  • WhatsApp Messaging Profile -> Webhook URL -> https://<id>.ngrok.io/webhooks/whatsapp

Open http://localhost:3000 in a browser to see the dashboard. Register with your phone number, then message the WhatsApp number from your phone.

curl http://localhost:3000/health
# {"status":"ok","timestamp":"2026-11-18T12:00:00.000Z","whatsapp_number":"+447537128942"}

curl http://localhost:3000/api/products?category=Fruits%20%26%20Vegetables
# {"products":[{"name":"Organic Bananas","price":1.29,...}],"total":8}

The Dashboard

The 3-panel dashboard updates in real time via Server-Sent Events:

Left panel — Product Catalog. Nine categories, 70+ products with prices and dietary tags. Products highlight when a customer browses them via WhatsApp.

Center panel — WhatsApp Phone Mockup. A phone frame showing the live conversation between the customer and Mia. Typing indicators, tool activity, inbound and outbound messages all render in real time.

Right panel — Delivery Pipeline. Order cards with progress bars showing each stage: Confirmed -> Picking -> Packed -> Out for Delivery -> Delivered. An "Advance" button lets you manually push orders through the pipeline for live demos, with WhatsApp notifications sent at each step.

Going to Production

This example uses SQLite for local development. For production:

  • Database — The app includes a Telnyx Edge SQL adapter in db.js for serverless deployment. Same queries, different connection string.
  • Webhook security — Add signature verification for inbound WhatsApp webhooks
  • Rate limiting — Protect the webhook endpoint from replay attacks
  • Conversation limits — Cap conversation history sent to AI Inference to stay within the model's context window
  • Error recovery — If AI Inference fails, send a "let me try that again" message rather than leaving the customer waiting
  • Multi-instance — Move session state to Redis if running multiple server instances behind a load balancer

Frequently Asked Questions

Do I need a WhatsApp Business Account to test this? Yes. You need a Telnyx account connected to a WABA (WhatsApp Business Account) with an approved phone number and a messaging profile configured with your webhook URL.

Can customers order items not in the catalog? Yes. The AI agent creates ad-hoc products when a customer asks for something not in the seed catalog. This makes the demo flexible — any grocery item is orderable.

How does voice note ordering work? The customer sends a WhatsApp voice note. The webhook detects audio, downloads it, sends it to Telnyx Audio Transcription (Whisper model), and feeds the transcript to the AI agent as text. The agent parses the items and calls add_to_cart for each one.

Why two API keys? The main Telnyx API key powers AI Inference, Number Lookup, and Audio Transcription. The WhatsApp API key comes from the WABA-connected account for sending and receiving WhatsApp messages. This is a consequence of the two-account setup, not a product limitation.

How does the delivery pipeline work? Orders progress through five stages: Confirmed -> Picking -> Packed -> Out for Delivery -> Delivered. The dashboard has an Advance button that moves the order to the next stage and sends a WhatsApp notification to the customer at each step.

Can the AI handle substitutions? Yes. When an item is out of stock, the AI sends a WhatsApp button message offering alternatives. The customer taps Accept, Reject, or Skip, and the AI processes the response.

How does the dashboard update in real time? Server-Sent Events (SSE) push events to the browser. Every AI tool call, WhatsApp message, order update, and cart change triggers an SSE event that the dashboard JavaScript handles to update the UI.

Ready to build with low-latency voice AI?

Join developers building the future of real-time conversations