Voice AI

Build a Voice Assistant That Remembers Every Caller

Most voice assistants begin every call with a blank slate.

The caller gives their name again. They explain why they called last time. They repeat preferences that the business already knows. Any useful context collected during the previous conversation sits somewhere else, disconnected from the live call.

The per-caller-intake sample demonstrates another approach: one durable intake dossier per caller.

Before a conversation begins, the assistant retrieves that caller's context from a Stateful Actor. During the call, it can use personalized dynamic variables and a freshly encrypted portal credential. After the conversation, it files a structured summary back into the same dossier.

The result is a voice assistant that can remember the last interaction without rebuilding the caller's history on every request.

What does the application do?

The sample models a fictional healthcare intake workflow. A returning caller can receive:

  • A personalized greeting with their name and preferred provider
  • The date of their most recent visit
  • Relevant account context, such as a balance
  • A fresh encrypted portal credential for the current conversation

When the call ends, a post-conversation tool files the visit reason, summary, and next step into the caller's dossier.

The example uses healthcare-style data to make the lifecycle concrete, but the architecture also applies to customer support, property management, logistics, financial services, and other workflows where conversations should build on earlier interactions.

How does one actor per caller work?

The central design decision is the actor key.

Telnyx sends an assistant.initialization event before the conversation begins. Its payload includes telnyx_end_user_target, which identifies the end user for the interaction.

The edge function normalizes that value and uses it to address one Stateful Actor:

const phoneDigits = normalizePhoneDigits(
  event.telnyx_end_user_target,
);

const stub = env.DOSSIERS.idFromName(phoneDigits);
const vars = await stub.handleInitialization(lookbackDays);

Calls associated with the same normalized number reach the same actor. That actor owns the caller's identity, preferences, visit history, and filing state.

This is useful because the ownership boundary is explicit. The AI Assistant manages the conversation, while the actor manages durable application context.

A phone number is a routing key in this sample, not proof of identity. Production applications should use telnyx_end_user_target_verified and their own authentication policy before returning sensitive data.

What data lives inside the actor?

Each IntakeDossier actor has a private embedded SQL database. The sample creates three tables:

CREATE TABLE IF NOT EXISTS visits (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  at TEXT NOT NULL,
  reason TEXT NOT NULL,
  summary TEXT,
  next_step TEXT
);

CREATE TABLE IF NOT EXISTS identity (
  patient_name TEXT NOT NULL,
  balance_due TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS prefs (
  provider TEXT
);

SQL is a good fit for visit history because the application needs ordered records and targeted queries. The actor does not have to load and rewrite one growing JSON object every time a conversation ends.

The sample seeds fictional demo values for Sarah, Dr. Lee, and a zero balance. A production implementation would populate these tables from an authenticated onboarding flow or an existing system of record.

The actor also maintains small pieces of durable state:

{
  lastVisitAt: string | null;
  lastFiledVisitId: number | null;
}

The relational history stays in SQL, while the actor state keeps the latest lifecycle markers readily available.

How is the initialization webhook secured?

The assistant calls:

POST /webhook/initialization

The handler reads the raw request body and verifies the Telnyx Ed25519 signature before parsing or trusting the event. Verification uses the timestamp, signature headers, raw body, and the public key stored in TELNYX_PUBLIC_KEY.

const rawBody = await req.text();
const signatureOk = await verifyTelnyxSignature(
  rawBody,
  req.headers,
  publicKey,
);

if (!signatureOk) {
  return errorResponse("invalid webhook signature", 401);
}

The verifier also rejects timestamps outside a five-minute window to reduce replay risk. If the public key is missing, the endpoint fails closed rather than accepting an unverified request.

After verification, the handler requires an assistant.initialization event and extracts telnyx_end_user_target from its payload.

What does the actor return to the assistant?

The actor reads the caller's identity, preferences, and most recent visit. It then returns two groups of runtime values:

{
  "dynamic_variables": {
    "patient_name": "Sarah",
    "provider": "Dr. Lee",
    "last_visit": "Jun 12, 2026",
    "balance_due": "$0.00"
  },
  "encrypted_dynamic_variables": {
    "portal_token": "AXN2aWV3X3Rva2VuX2NpcGhlcnRleHQ..."
  }
}

The regular dynamic variables can personalize the assistant's greeting and instructions. The encrypted value follows a different path because credentials should not be placed in ordinary prompt variables.

Telnyx uses a 1.5-second default timeout for a dynamic variables webhook. The example configures dynamic_variables_webhook_timeout_ms to 8000 to allow additional room for initialization and cold starts.

How is the per-caller credential encrypted?

For every initialization, the actor generates a fresh random 32-byte token. It encrypts that token using AES-256-GCM through the Web Crypto API:

const rawToken = base64UrlEncode(
  crypto.getRandomValues(new Uint8Array(32)),
);

const encKey = await this.getEncKey();
const portalToken = await encryptPortalToken(rawToken, encKey);

The encryption function creates a 12-byte nonce and returns this format:

base64url(nonce || ciphertext+authentication-tag)
const key = await crypto.subtle.importKey(
  "raw",
  keyBytes,
  { name: "AES-GCM" },
  false,
  ["encrypt"],
);

const nonce = crypto.getRandomValues(new Uint8Array(12));
const ciphertext = await crypto.subtle.encrypt(
  { name: "AES-GCM", iv: nonce },
  key,
  new TextEncoder().encode(plaintext),
);

The same 32-byte key must be stored as both the Edge secret PORTAL_ENC_KEY and the Telnyx integration secret portal_enc_key. The sample returns only the encrypted value in the webhook response. The assistant can reference the credential through the per-caller credentials integration rather than putting its plaintext value into model context.

Both copies of the key must be rotated together.

How is the conversation filed afterward?

Post-conversation processing invokes:

POST /webhook/post-conversation

The endpoint requires a bearer token matching DOSSIER_WEBHOOK_AUTH. A missing secret fails closed, and an incorrect token receives a 401 response.

The structured payload contains the caller target and the outcome of the conversation:

{
  "telnyx_end_user_target": "+15551234567",
  "visit_reason": "follow-up",
  "follow_up": "Discussed hearing aid maintenance.",
  "next_step": "rescheduled to Thursday"
}

The handler derives the same actor ID from telnyx_end_user_target and calls fileVisitSummary() on that caller's dossier.

The actor checks for an existing row with the same reason and next step before inserting:

const existing = this.ctx.storage.sql
  .exec<{ id: number }>(
    `SELECT id FROM visits
     WHERE reason = ? AND next_step = ?
     ORDER BY id DESC LIMIT 1`,
    reason,
    nextStep,
  )
  .toArray();

if (existing.length > 0) {
  return { filed: false, visit_id: existing[0].id };
}

This gives the demo retry protection for identical filing attempts. A production integration should prefer a stable event or tool invocation ID as its idempotency key when one is available.

After a successful insert, the actor updates lastVisitAt and lastFiledVisitId. The next initialization reads the newly filed history.

How do you run the sample?

You need Node.js 20 or newer, Telnyx Edge CLI 0.2.2 or newer, and a Telnyx API key.

git clone https://github.com/team-telnyx/telnyx-code-examples.git
cd telnyx-code-examples/per-caller-intake
npm install
npx tsc --noEmit
npm run smoke

The smoke test covers the security-sensitive paths before deployment:

  • AES-GCM encryption and decryption
  • Valid Ed25519 signatures
  • Modified request bodies
  • Stale timestamps
  • Incorrect public keys
  • Initialization event parsing

Generate the credential key:

KEY=$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '=')
telnyx-edge secrets add PORTAL_ENC_KEY "$KEY"

Register the same value as the portal_enc_key integration secret. Then retrieve your Telnyx public key and store it at the edge:

PUBLIC_KEY=$(curl -s \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  https://api.telnyx.com/v2/public_key \
  | jq -r '.data.public')

telnyx-edge secrets add TELNYX_PUBLIC_KEY "$PUBLIC_KEY"

Create a separate bearer token for the filing webhook and store matching copies as the Edge secret DOSSIER_WEBHOOK_AUTH and the integration secret dossier_webhook_auth.

Deploy the function:

telnyx-edge ship

Use the generated function URL for the initialization webhook and filing tool. The full assistant configuration is included in the sample's GUIDE.md.

What should change before production?

The sample makes the architecture visible, but a real deployment needs additional policy and controls:

  • Authenticate callers before returning sensitive context
  • Treat a phone number as a routing key, not automatic proof of identity
  • Protect or remove the development-only /dossier/:phone route
  • Define data retention, deletion, and consent policies
  • Replace demo identity seeding with an authenticated data source
  • Use a stable idempotency key for post-conversation writes
  • Rotate both copies of the credential encryption key together
  • Review the privacy and regulatory requirements for your industry

Where else can this pattern be used?

The reusable pattern is broader than patient intake:

  1. Derive a stable actor ID from the interaction.
  2. Load only the context needed for the current conversation.
  3. Encrypt credentials that should not enter model context.
  4. Store the structured outcome after the conversation.
  5. Route the next interaction back to the same actor.

A support line can remember open cases and previous troubleshooting. A property-management assistant can retain unit details and maintenance history. A delivery assistant can remember preferred drop-off instructions. A financial-services assistant can load account context after a separate authentication step.

The assistant handles the conversation. The actor gives that conversation durable application memory.

Ready to build with low-latency voice AI?

Join developers building the future of real-time conversations