Voice AI

Post-Call QA Scoring: Durable Agent Quality Profiles with Telnyx Decision Models

Every support organization runs the same uncomfortable math: thousands of calls per week, a handful of supervisors, and a QA process that samples a sliver of it. In regulated industries — banking, healthcare, insurance — the stakes compound. A single missed compliance disclosure can mean regulatory fines, customer churn, or reputational damage. And even when a call does get graded, the grade is a snapshot: one call, one score, no memory.

Post-call QA scoring is the practice of automatically grading a completed call transcript against quality and compliance criteria — and, just as importantly, tracking those grades over time so coaching decisions rest on trends, not anecdotes. Coaching based on a single call is guesswork. Coaching based on a durable trend is precision.

This post walks through a working implementation built on Telnyx AI Communications Infrastructure: it grades every support call with the Decision Models API, keeps a durable per-agent quality profile, flags coaching needs from rolling trends, and texts a daily digest to the team lead. The full source lives in the telnyx-code-examples repo.

What the App Does

The scenario is a regional bank's customer support desk. When a support call ends, the system:

  1. Receives the transcript. A call-conversation-ended webhook arrives with the transcript embedded. If it isn't embedded, a transcription-saved webhook delivers the finalized transcript as a fallback.
  2. Routes to a durable actor. One QAAgent actor exists per support agent, addressed via idFromName(agentId) — born on first delivery, self-provisioning.
  3. Grades the call. The actor calls the Telnyx Decision Models API (POST /v2/ai/typesafe/v1/systemone) and gets three structured signals in one shared-state call: a pass/fail choice with failing category, a hard compliance breach score (noul), and a 0–5 quality score.
  4. Persists the result. Every grade is appended to the actor's private SQL history — a scores table keyed by call_id, plus a breaches table.
  5. Tracks the trend. The actor recomputes a 5-call rolling average. Below the coaching floor, the agent is flagged for coaching; when performance recovers, the flag auto-clears.
  6. Escalates breaches. If the compliance breach score exceeds 0.8, the call is flagged for manager review regardless of pass/fail, and an immediate SMS alert goes to the team lead.
  7. Reports daily. At 17:00 UTC, the actor texts a one-line digest of its agent's standing to the team lead.

The actor survives pod reboots, platform restarts, and weeks of calls — because a trend needs history, and history needs durability.

How It Works

The webhook front door

A thin worker receives Telnyx webhooks and routes each event to the right actor. The routing is one line of consequence: idFromName(agentId) deterministically maps an agent ID to a durable actor.

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);

    if (request.method === "POST" && url.pathname === "/webhook/call-conversation-ended") {
      const event = await request.json();
      const agentId = event.data.agent_id;

      // One QAAgent per support agent — born on first delivery
      const stub = env.QA_AGENT.get(env.QA_AGENT.idFromName(agentId));
      await stub.recordCallEnded(event.call_id, event.transcript, agentId);
      return new Response("ok");
    }

    // Fallback: finalized transcript arrives here if it wasn't
    // embedded in the ended payload
    if (request.method === "POST" && url.pathname === "/webhook/transcription-saved") {
      const event = await request.json();
      const stub = env.QA_AGENT.get(env.QA_AGENT.idFromName(event.data.agent_id));
      await stub.recordTranscription(event.call_id, event.transcript);
      return new Response("ok");
    }

    return new Response("Not found", { status: 404 });
  },
};

Snippets below are abridged from the repo for readability — clone it for the complete implementation.

Grading with Decision Models

The grading call is the heart of the system. Instead of prompting a model and parsing free-text output, the Decision Models API accepts a typed questions object and returns structured fields — choice, noul, and score — in a single shared-state call:

private async buildDecisionRequest(transcript: string) {
  const res = await fetch("https://api.telnyx.com/v2/ai/typesafe/v1/systemone", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${this.env.SECRETS.get("TELNYX_API_KEY")}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      input: transcript,
      questions: {
        passed: {
          type: "choice",
          choices: ["pass", "fail"],
          description: "Did the agent meet QA standards on this call?",
        },
        failing_category: {
          type: "choice",
          choices: ["compliance", "empathy", "accuracy", "resolution", "none"],
          description: "If failed, which category? 'none' if passed.",
        },
        compliance_breach: {
          type: "noul",
          description:
            "0-1 likelihood of a hard compliance breach (missed disclosure, unauthorized action).",
        },
        quality: { type: "score", min: 0, max: 5, description: "Overall call quality." },
      },
    }),
  });

  if (!res.ok) throw new Error(`Decision Models responded ${res.status}`);
  return res.json(); // → { choice, noul, score }
}

Structured outputs matter here for a practical reason: the downstream logic — trend math, breach thresholds, digest lines — consumes typed values, not scraped prose.

Exactly-once grading

Webhooks retry. Without protection, a single call could be graded twice and quietly pollute the trend. The sample defends in two layers:

async recordCallEnded(callId: string, transcript: string, agentId: string) {
  // Layer 1: fast-path dedup check
  if (await this.findScore(callId)) return;

  this.ctx.storage.sql.exec(
    `INSERT INTO scores (call_id, agent_id, transcript, status)
     VALUES (?, ?, ?, 'pending')`,
    callId, agentId, transcript
  );

  // Layer 2: stable schedule id — redeliveries collapse onto the same task
  this.schedule(0, "gradeCall", { callId, attempt: 0 }, `grade:${callId}`);

  this.scheduleDigest(); // bootstrap the daily digest if not armed
}

The scores table uses call_id as its PRIMARY KEY, so even a race between the two webhook paths resolves to one row.

Backoff that respects the API

Model APIs rate-limit. The grading task retries 429/5xx responses with 10s → 30s → 60s backoff, honoring Retry-After when present. If retries are exhausted, the call is marked status="ungraded" with a last_error — never silently dropped:

async gradeCall({ callId, attempt }: { callId: string; attempt: number }) {
  const row = await this.findScore(callId);
  if (!row || row.status === "graded") return;

  try {
    const decision = await this.buildDecisionRequest(row.transcript);

    this.ctx.storage.sql.exec(
      `UPDATE scores SET status = 'graded', choice = ?, noul = ?, score = ?
       WHERE call_id = ?`,
      decision.choice, decision.noul, decision.score, callId
    );

    await this.recomputeTrend();

    if (decision.noul > 0.8) {
      this.ctx.storage.sql.exec(
        `INSERT INTO breaches (call_id, agent_id, noul, reviewed) VALUES (?, ?, ?, 0)`,
        callId, row.agent_id, decision.noul
      );
      await this.alertBreach(callId, decision.noul); // immediate SMS
    }
  } catch (err) {
    if (attempt < 2) {
      const delay = [10, 30, 60][attempt];
      this.schedule(delay, "gradeCall", { callId, attempt: attempt + 1 }, `grade:${callId}`);
    } else {
      this.ctx.storage.sql.exec(
        `UPDATE scores SET status = 'ungraded', last_error = ? WHERE call_id = ?`,
        String(err), callId
      );
    }
  }
}

Trends and coaching flags

After each grade, recomputeTrend() pulls the last five graded calls, computes the rolling average, and compares it to the coaching floor:

private async recomputeTrend() {
  const rows = this.ctx.storage.sql
    .exec(`SELECT score FROM scores WHERE status = 'graded'
           ORDER BY created_at DESC LIMIT 5`)
    .toArray();

  if (rows.length === 0) return;

  const rolling = rows.reduce((s, r) => s + r.score, 0) / rows.length;
  const floor = Number(this.env.SECRETS.get("COACHING_FLOOR") ?? 3.5);

  this.setState({
    ...this.state,
    rolling: Math.round(rolling * 100) / 100,
    flagged: rolling < floor, // flag below the floor…
    worstCategory: this.computeWorstCategory(),
  }); // …and auto-clear on recovery
}

Note what the flag is not: it isn't a one-off alert. It's durable state on the actor — rolling, flagged, worstCategory — recomputed on every grade and cleared automatically when the average recovers. The trend is the product.

The daily digest

At 17:00 UTC, each actor texts its own one-liner to the team lead via the TELNYX binding's send-a-message — no external SMS provider, no credential management:

async runDigest() {
  const line = this.buildDigestLine(); // "Agent A-114: rolling 4.2, trending up, 0 breaches"
  await this.env.TELNYX.send({
    to: this.env.SECRETS.get("TEAM_LEAD_E164"),
    text: line,
  });

  // Re-arm for tomorrow — the chain survives crashes and redeploys
  this.schedule(this.msUntilNextDigestHour(), "runDigest", {}, "digest");
}

The digest task re-arms itself after every run under the stable digest id. If the platform restarts mid-chain, the next tick still fires.

Setup

  1. Clone the repo and install dependencies.
  2. Declare secrets in telnyx.toml — all configuration flows through Telnyx [[secrets]], read at runtime via this.env.SECRETS.get():
[[secrets]]
name = "TELNYX_API_KEY"    # Decision Models + Messaging

[[secrets]]
name = "TEAM_LEAD_E164"    # digest recipient, E.164 format

[[secrets]]
name = "TELNYX_NUMBER"     # sending number for digests and alerts

[[secrets]]
name = "DIGEST_HOUR_UTC"   # default: 17

[[secrets]]
name = "COACHING_FLOOR"    # default: 3.5
  1. Deploy the worker to the Telnyx Edge Runtime.
  2. Wire the webhooks. Point your Call Control configuration's call-conversation-ended and transcription-saved events at the deployed /webhook/... URLs.
  3. Test without making a call. Hit /demo/trigger with an agent ID to push a synthetic call through the full pipeline: webhook → actor → grading → trend → digest.

Conclusion

The interesting part of this sample isn't the grading — it's the durability. One actor per agent, private SQL per actor, exactly-once grading, self-re-arming schedules: these are the pieces that turn "AI graded a call" into "the system knows whether this agent is trending down over three weeks."

Natural extensions fall out of the same shape: per-category trend breakdowns using worstCategory, weekly rollups across the whole desk, or a supervisor dashboard reading the same actor state. Start from the repo and build the version your QA team actually needs: post-call-qa-scoring.

Ready to build with low-latency voice AI?

Join developers building the future of real-time conversations