Build a Voice Agent That Actually Holds the Call

The first time you get an AI agent to answer a phone call, it feels magical.

Then the real problem shows up.

Answering the call is only the first step. A useful voice agent needs to keep listening, decide when the caller has finished speaking, run the model, speak back, and then return to listening again. It also needs to remember the call state while all of this is happening.

That is the pattern this example demonstrates.

edge-voice-agent-holds-call is a TypeScript example that runs on Telnyx Edge Compute with the Telnyx Agent SDK. It answers an inbound call, speaks a greeting, starts streaming transcription, turns final transcripts into LLM responses, speaks the response back with TTS, and loops until the caller hangs up.

Code: https://github.com/team-telnyx/telnyx-code-examples/tree/main/edge-voice-agent-holds-call

What we are building

The app is a voice agent that can hold a live phone conversation.

Telnyx Call Control sends webhooks into an Edge Compute function. The function dispatches each call to a VoiceAgent actor keyed by call_control_id. That actor stores the call phase, turn count, transcript history, and assistant replies.

At a high level, the loop looks like this:

Inbound call
  -> answer
  -> speak greeting
  -> start transcription
  -> final caller transcript
  -> stop transcription
  -> run LLM turn
  -> speak reply
  -> start transcription again

That loop is the product.

The agent is not just responding to one webhook. It is coordinating the call from one phase to the next.

Why this belongs at the edge

Voice applications are different from most HTTP apps.

When someone is on a live call, latency is part of the experience. Every extra hop between the phone call, transcription, model response, and TTS reply shows up as dead air.

This example keeps the control loop close to the communications path:

• Call Control handles answer, speak, transcription start/stop, and hangup
• Edge Compute receives the webhooks and dispatches state transitions
• the Agent SDK stores durable per-call history
• Telnyx AI Inference generates each response through the [telnyx] binding
• TTS speaks the reply back into the same call

That is the interesting architecture: telephony, AI, and state are composed in one edge application.

The call lifecycle

The webhook router handles a few Call Control events.

When call.initiated arrives, the app records the call start in the actor and answers the call.

When call.answered arrives, the app speaks the greeting:

Hi, this is an AI voice agent. What can I help you with today?

When call.speak.ended arrives, the app starts streaming transcription on the inbound track. That means it is listening to the caller.

When a final call.transcription event arrives, the app stops transcription, stores the caller's speech in message history, asks the model for a short conversational answer, and speaks the reply back through Call Control TTS.

Then call.speak.ended fires again and the loop starts another listen turn.

The Agent SDK role

The VoiceAgent class extends the Agent SDK Agent class.

That gives the app durable primitives for a live call:

• this.messages.add() stores caller and assistant turns
• this.messages.toOpenAI() formats the history for AI Inference
• this.setState() and this.getState() store call phase, caller/callee, turn count, and timestamps
• this.env.TELNYX.ai.openai.chat.createCompletion() calls Telnyx AI Inference through the pre-authenticated [telnyx] binding

The inference call uses the OpenAI-compatible shape:

const completion = await this.env.TELNYX.ai.openai.chat.createCompletion({
  model,
  messages: [{ role: "system", content: SYSTEM_PROMPT }, ...history],
  max_tokens: 200,
  temperature: 0.5,
});

The system prompt asks the assistant to stay brief and natural because this is a live call, not a chat window.

That detail matters. A phone answer that looks fine in text can feel painfully long when spoken aloud.

Why streaming transcription

The sample uses streaming transcription instead of treating the call as a one-shot speech gather.

That lets the app own the conversation loop. The caller speaks, final transcript events arrive, the agent runs its own LLM turn, and the app speaks its own TTS response.

This gives you room to add application logic between turns:

• route certain intents to tools
• cap the number of turns
• hand off to a human
• track cost per call
• personalize based on caller state
• store a transcript summary after hangup

It also keeps the model call inside your own agent code instead of handing the whole conversation loop to a hosted assistant configuration.

Handling real call behavior

The sample includes a few details that make it feel more like a real voice app:

• a silence timeout that reprompts with "Are you still there?"
• a second timeout path that can hang up if there is still no response
• basic barge-in handling when speech arrives while the agent is replying
• filtering to avoid treating the agent's own TTS as caller speech
• a debug state endpoint for inspecting call phase and message count

Those details are easy to skip in a demo, but they matter once someone actually calls the number.

Voice agents are state machines. The better your state transitions are, the less awkward the call feels.

Running it

Install dependencies and deploy:

npm install
telnyx-edge ship

Set the required Call Control API key as an Edge secret:

telnyx-edge secret set TELNYX_API_KEY your_telnyx_api_key

Then point your Call Control application webhook to:

https://edge-voice-agent-holds-call-<id>.telnyxcompute.com/webhooks/voice

Call the Telnyx number. The agent should answer, greet, listen, respond, and continue the conversation loop until hangup.

Where this pattern goes next

This sample is deliberately small, but it is the shape of a lot of useful voice-agent work.

You could adapt it for:

• appointment scheduling
• order status lookup
• support triage
• lead qualification
• dispatch intake
• billing questions
• after-hours coverage

Before production, I would add webhook signature verification, stronger idempotency around Call Control actions, human handoff, transcript redaction, turn limits, call-duration limits, and latency monitoring across STT, inference, and TTS.

But the core idea is already here.

If your AI agent is going to answer the phone, it needs more than a prompt. It needs a call loop, durable state, and the ability to keep listening.