Voice AI

Schema-First Conversation Mining: Build a QA Scorecard with Telnyx AI Insights

Every AI phone call your assistant handles ends the same way: with a summary. Free text. Prose. And prose is where analytics goes to die — someone reads it, hand-tags a spreadsheet, and six months later a regulator asks, "show me every call where the disclosure line was missed," and nobody can answer without an archaeology project.

insights-pipeline is a sample that fixes this at the source. It uses Telnyx AI Insights to turn call conversations into typed records — sentiment scores, compliance flags, escalation reasons — validated against JSON schemas you define, delivered by webhook or poll, and rendered into a QA scorecard with zero prose parsing.

In this post we'll walk the whole pipeline: templates, groups, assistant wiring, a live call, and the scorecard.

What the App Does

The demo scenario is a mid-sized outpatient clinic running a patient-services phone line. Every call is a liability surface: a missed identity check, a skipped recording disclosure, an ignored escalation request. The QA lead currently reads free-text call summaries and hand-tags them in a spreadsheet — slow, inconsistent, and impossible to audit.

The sample replaces that with a four-layer pipeline:

  1. Insight templates — three schemas (sentiment-v1, compliance-v1, escalation-v1) declaring exactly what the model must extract.
  2. Insight group — the three templates bundled into qa-scorecard-v1, so every conversation is analyzed once with all three concerns applied.
  3. Assistant wiring — the group attached to the assistant via insight_settings, once.
  4. Delivery + render — a demo call is placed, insights arrive by webhook (with a polling fallback), and scorecard.py renders typed values: sentiment distribution, compliance-flag rates, escalation breakdown.

How It Works

Layer 1: Templates are contracts

The QA scorecard is born the moment an insight template declares its json_schema. That schema is the shape of a record the model is contractually obligated to produce — not a suggestion, a contract.

COMPLIANCE_TEMPLATE = {
    "name": "compliance-v1",
    "instructions": (
        "Review the conversation transcript. Flag every compliance "
        "requirement that was not satisfied during the call."
    ),
    "json_schema": {
        "type": "object",
        "properties": {
            "verified_identity": {"type": "boolean"},
            "compliance_flags": {
                "type": "array",
                "items": {"type": "string"},
            },
        },
        "required": ["verified_identity", "compliance_flags"],
    },
}

resp = requests.post(
    f"{API_BASE}/ai/conversations/insights",
    headers=HEADERS,
    json=COMPLIANCE_TEMPLATE,
)
insight_id = resp.json()["data"]["id"]

The sample creates three templates. sentiment-v1 extracts positivity, frustration, a sentiment enum, and an emotions array. escalation-v1 extracts escalation_reason and whether the agent performed the escalation.

Before creating anything, the script lists existing templates. That's idempotency plus a drift check: if a template with the same name already exists but its schema has changed since you last deployed, you find out at deploy time — not three weeks later in your dashboard.

Layer 2: One group, three concerns

Templates are bundled into an insight group, and each template is assigned to it:

group = requests.post(
    f"{API_BASE}/ai/conversations/insight-groups",
    headers=HEADERS,
    json={
        "name": "qa-scorecard-v1",
        "description": "Sentiment + compliance + escalation QA",
        "webhook": WEBHOOK_URL,
    },
).json()["data"]

for insight_id in (sentiment_id, compliance_id, escalation_id):
    requests.post(
        f"{API_BASE}/ai/conversations/insight-groups/{group['id']}"
        f"/insights/{insight_id}/assign",
        headers=HEADERS,
    )

Group creation is find-or-create by name, and after assignment the script re-reads the group to verify membership. Trust the verification, not the assumption.

Layer 3: Attach once, analyze forever

The group attaches to the assistant through insight_settings:

requests.post(
    f"{API_BASE}/ai/assistants/{assistant_id}",
    headers=HEADERS,
    json={"insight_settings": {"insight_group_id": group_id}},
)

One GET /ai/assistants/{assistant_id} confirms insight_settings.insight_group_id is attached. From here, every call conversation this assistant handles is analyzed once, with all three templates applied.

Layer 4: Delivery, with a fallback that tells the truth

The demo call is placed through the assistant's scheduled events endpoint (channel phone_call), and the response carries the conversation_id the rest of the pipeline hangs off of.

Insight records land two ways. The happy path is the call.conversation.insights_generated webhook, received by a small Flask app:

@app.route("/webhooks/insights", methods=["POST"])
def insights_generated():
    event = request.get_json()
    conversation_id = event["data"]["payload"]["conversation_id"]
    append_to_store(conversation_id)  # conversations.json
    return "", 204

But the webhook receiver is a separate process, and separate processes get killed, redeployed, and forgotten. So the pipeline falls back to polling the same source of truth:

def poll_insights(conversation_id, timeout=300):
    deadline = time.time() + timeout
    while time.time() < deadline:
        records = requests.get(
            f"{API_BASE}/ai/conversations/{conversation_id}"
            f"/conversations-insights",
            headers=HEADERS,
        ).json()["data"]
        if all(r["status"] in ("completed", "failed") for r in records):
            return records
        time.sleep(5)
    raise TimeoutError("insights did not finish in time")

The poll fallback produces the identical scorecard. That's the point: the webhook is an optimization, not the system of record.

Record statuses are pending, in_progress, completed, and failed, and the scorecard treats them honestly:

  • A record still in_progress is a pending row, not a failure.
  • A completed record missing a required field is flagged by conversation ID and excluded from aggregates rather than silently poisoning the numbers.
def render_scorecard(records):
    completed = [r for r in records if r["status"] == "completed"]
    pending = [r for r in records if r["status"] == "in_progress"]

    invalid = [r for r in completed if missing_required_fields(r)]
    valid = [r for r in completed if r not in invalid]

    return {
        "sentiment_distribution": tally(valid, "sentiment"),
        "compliance_flag_rate": rate(valid, "compliance_flags"),
        "escalation_breakdown": tally(valid, "escalation_reason"),
        "pending": [r["conversation_id"] for r in pending],
        "excluded": [r["conversation_id"] for r in invalid],
    }

When the scorecard renders, it renders typed values — sentiment distribution, compliance-flag rates, escalation breakdown — with zero prose parsing. Nobody reads a summary. Nobody tags a spreadsheet.

Setup

You'll need a Telnyx API key, a voice-enabled number, and (optionally) an existing assistant ID. Set the environment variables:

export TELNYX_API_KEY="KEY123..."
export TELNYX_API_BASE="https://api.telnyx.com/v2"  # default
export ASSISTANT_ID=""          # optional; created if unset
export FROM_NUMBER="+15551234567"
export TO_NUMBER="+15557654321"
export WEBHOOK_URL="https://your-host.example/webhooks/insights"

Then run the layers in order:

python create_templates.py   # Layer 1: three insight templates
python create_group.py       # Layer 2: group + assignments
python wire_assistant.py     # Layer 3: attach insight_settings
python place_call.py         # Layer 4: demo call + poll
python webhook_receiver.py   # optional: real-time delivery
python scorecard.py          # render the QA scorecard

scorecard.py accepts --date and --conversation-id filters for drilling into individual calls.

Conclusion

Structured insights are the analytics-grade half of AI Insights — the half that feeds dashboards. Telnyx exposes them as first-class, composable primitives: declare a json_schema, bundle templates into a group, attach the group to an assistant once, and every call conversation is mined into typed records. The webhook closes the loop for real-time QA consoles; the polling endpoint is the source of truth when the receiver isn't up.

This is what AI Communications Infrastructure looks like in practice: the model's output is a typed record on day one, not prose you regex later. Clone the sample, wire your own templates, and build the scorecard your QA team actually needs.

Ready to build with low-latency voice AI?

Join developers building the future of real-time conversations