Every support organization runs the same uncomfortable math: thousands of calls per week, a handful of supervisors, and a QA process that samples a sliver of it. In regulated industries — banking, healthcare, insurance — the stakes compound. A single missed compliance disclosure can mean regulatory fines, customer churn, or reputational damage. And even when a call does get graded, the grade is a snapshot: one call, one score, no memory.
Post-call QA scoring is the practice of automatically grading a completed call transcript against quality and compliance criteria — and, just as importantly, tracking those grades over time so coaching decisions rest on trends, not anecdotes. Coaching based on a single call is guesswork. Coaching based on a durable trend is precision.
This post walks through a working implementation built on Telnyx AI Communications Infrastructure: it grades every support call with the Decision Models API, keeps a durable per-agent quality profile, flags coaching needs from rolling trends, and texts a daily digest to the team lead. The full source lives in the telnyx-code-examples repo.
What the App Does
The scenario is a regional bank's customer support desk. When a support call ends, the system:
- Receives the transcript. A
call-conversation-endedwebhook arrives with the transcript embedded. If it isn't embedded, atranscription-savedwebhook delivers the finalized transcript as a fallback. - Routes to a durable actor. One
QAAgentactor exists per support agent, addressed viaidFromName(agentId)— born on first delivery, self-provisioning. - Grades the call. The actor calls the Telnyx Decision Models API (
POST /v2/ai/typesafe/v1/systemone) and gets three structured signals in one shared-state call: a pass/failchoicewith failing category, a hard compliance breach score (noul), and a 0–5 qualityscore. - Persists the result. Every grade is appended to the actor's private SQL history — a
scorestable keyed bycall_id, plus abreachestable. - Tracks the trend. The actor recomputes a 5-call rolling average. Below the coaching floor, the agent is flagged for coaching; when performance recovers, the flag auto-clears.
- Escalates breaches. If the compliance breach score exceeds 0.8, the call is flagged for manager review regardless of pass/fail, and an immediate SMS alert goes to the team lead.
- Reports daily. At 17:00 UTC, the actor texts a one-line digest of its agent's standing to the team lead.
The actor survives pod reboots, platform restarts, and weeks of calls — because a trend needs history, and history needs durability.
How It Works
The webhook front door
A thin worker receives Telnyx webhooks and routes each event to the right actor. The routing is one line of consequence: idFromName(agentId) deterministically maps an agent ID to a durable actor.
export default {
async fetch(request: Request, env: Env): Promise<Response> {
const url = new URL(request.url);
if (request.method === "POST" && url.pathname === "/webhook/call-conversation-ended") {
const event = await request.json();
const agentId = event.data.agent_id;
// One QAAgent per support agent — born on first delivery
const stub = env.QA_AGENT.get(env.QA_AGENT.idFromName(agentId));
await stub.recordCallEnded(event.call_id, event.transcript, agentId);
return new Response("ok");
}
// Fallback: finalized transcript arrives here if it wasn't
// embedded in the ended payload
if (request.method === "POST" && url.pathname === "/webhook/transcription-saved") {
const event = await request.json();
const stub = env.QA_AGENT.get(env.QA_AGENT.idFromName(event.data.agent_id));
await stub.recordTranscription(event.call_id, event.transcript);
return new Response("ok");
}
return new Response("Not found", { status: 404 });
},
};
Snippets below are abridged from the repo for readability — clone it for the complete implementation.
Grading with Decision Models
The grading call is the heart of the system. Instead of prompting a model and parsing free-text output, the Decision Models API accepts a typed questions object and returns structured fields — choice, noul, and score — in a single shared-state call:
private async buildDecisionRequest(transcript: string) {
const res = await fetch("https://api.telnyx.com/v2/ai/typesafe/v1/systemone", {
method: "POST",
headers: {
Authorization: `Bearer ${this.env.SECRETS.get("TELNYX_API_KEY")}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: transcript,
questions: {
passed: {
type: "choice",
choices: ["pass", "fail"],
description: "Did the agent meet QA standards on this call?",
},
failing_category: {
type: "choice",
choices: ["compliance", "empathy", "accuracy", "resolution", "none"],
description: "If failed, which category? 'none' if passed.",
},
compliance_breach: {
type: "noul",
description:
"0-1 likelihood of a hard compliance breach (missed disclosure, unauthorized action).",
},
quality: { type: "score", min: 0, max: 5, description: "Overall call quality." },
},
}),
});
if (!res.ok) throw new Error(`Decision Models responded ${res.status}`);
return res.json(); // → { choice, noul, score }
}
Structured outputs matter here for a practical reason: the downstream logic — trend math, breach thresholds, digest lines — consumes typed values, not scraped prose.
Exactly-once grading
Webhooks retry. Without protection, a single call could be graded twice and quietly pollute the trend. The sample defends in two layers:
async recordCallEnded(callId: string, transcript: string, agentId: string) {
// Layer 1: fast-path dedup check
if (await this.findScore(callId)) return;
this.ctx.storage.sql.exec(
`INSERT INTO scores (call_id, agent_id, transcript, status)
VALUES (?, ?, ?, 'pending')`,
callId, agentId, transcript
);
// Layer 2: stable schedule id — redeliveries collapse onto the same task
this.schedule(0, "gradeCall", { callId, attempt: 0 }, `grade:${callId}`);
this.scheduleDigest(); // bootstrap the daily digest if not armed
}
The scores table uses call_id as its PRIMARY KEY, so even a race between the two webhook paths resolves to one row.
Backoff that respects the API
Model APIs rate-limit. The grading task retries 429/5xx responses with 10s → 30s → 60s backoff, honoring Retry-After when present. If retries are exhausted, the call is marked status="ungraded" with a last_error — never silently dropped:
async gradeCall({ callId, attempt }: { callId: string; attempt: number }) {
const row = await this.findScore(callId);
if (!row || row.status === "graded") return;
try {
const decision = await this.buildDecisionRequest(row.transcript);
this.ctx.storage.sql.exec(
`UPDATE scores SET status = 'graded', choice = ?, noul = ?, score = ?
WHERE call_id = ?`,
decision.choice, decision.noul, decision.score, callId
);
await this.recomputeTrend();
if (decision.noul > 0.8) {
this.ctx.storage.sql.exec(
`INSERT INTO breaches (call_id, agent_id, noul, reviewed) VALUES (?, ?, ?, 0)`,
callId, row.agent_id, decision.noul
);
await this.alertBreach(callId, decision.noul); // immediate SMS
}
} catch (err) {
if (attempt < 2) {
const delay = [10, 30, 60][attempt];
this.schedule(delay, "gradeCall", { callId, attempt: attempt + 1 }, `grade:${callId}`);
} else {
this.ctx.storage.sql.exec(
`UPDATE scores SET status = 'ungraded', last_error = ? WHERE call_id = ?`,
String(err), callId
);
}
}
}
Trends and coaching flags
After each grade, recomputeTrend() pulls the last five graded calls, computes the rolling average, and compares it to the coaching floor:
private async recomputeTrend() {
const rows = this.ctx.storage.sql
.exec(`SELECT score FROM scores WHERE status = 'graded'
ORDER BY created_at DESC LIMIT 5`)
.toArray();
if (rows.length === 0) return;
const rolling = rows.reduce((s, r) => s + r.score, 0) / rows.length;
const floor = Number(this.env.SECRETS.get("COACHING_FLOOR") ?? 3.5);
this.setState({
...this.state,
rolling: Math.round(rolling * 100) / 100,
flagged: rolling < floor, // flag below the floor…
worstCategory: this.computeWorstCategory(),
}); // …and auto-clear on recovery
}
Note what the flag is not: it isn't a one-off alert. It's durable state on the actor — rolling, flagged, worstCategory — recomputed on every grade and cleared automatically when the average recovers. The trend is the product.
The daily digest
At 17:00 UTC, each actor texts its own one-liner to the team lead via the TELNYX binding's send-a-message — no external SMS provider, no credential management:
async runDigest() {
const line = this.buildDigestLine(); // "Agent A-114: rolling 4.2, trending up, 0 breaches"
await this.env.TELNYX.send({
to: this.env.SECRETS.get("TEAM_LEAD_E164"),
text: line,
});
// Re-arm for tomorrow — the chain survives crashes and redeploys
this.schedule(this.msUntilNextDigestHour(), "runDigest", {}, "digest");
}
The digest task re-arms itself after every run under the stable digest id. If the platform restarts mid-chain, the next tick still fires.
Setup
- Clone the repo and install dependencies.
- Declare secrets in
telnyx.toml— all configuration flows through Telnyx[[secrets]], read at runtime viathis.env.SECRETS.get():
[[secrets]]
name = "TELNYX_API_KEY" # Decision Models + Messaging
[[secrets]]
name = "TEAM_LEAD_E164" # digest recipient, E.164 format
[[secrets]]
name = "TELNYX_NUMBER" # sending number for digests and alerts
[[secrets]]
name = "DIGEST_HOUR_UTC" # default: 17
[[secrets]]
name = "COACHING_FLOOR" # default: 3.5
- Deploy the worker to the Telnyx Edge Runtime.
- Wire the webhooks. Point your Call Control configuration's
call-conversation-endedandtranscription-savedevents at the deployed/webhook/...URLs. - Test without making a call. Hit
/demo/triggerwith an agent ID to push a synthetic call through the full pipeline: webhook → actor → grading → trend → digest.
Conclusion
The interesting part of this sample isn't the grading — it's the durability. One actor per agent, private SQL per actor, exactly-once grading, self-re-arming schedules: these are the pieces that turn "AI graded a call" into "the system knows whether this agent is trending down over three weeks."
Natural extensions fall out of the same shape: per-category trend breakdowns using worstCategory, weekly rollups across the whole desk, or a supervisor dashboard reading the same actor state. Start from the repo and build the version your QA team actually needs: post-call-qa-scoring.