How to Export SMS Conversation History to Cloud Storage with Chunked JSON and Agent SDK on Edge Compute

Export SMS conversation history from Edge SQL to Cloud Storage as chunked JSON files, with a 4-stage Agent SDK pipeline that handles 10k+ messages — and texts you when the export is done. All on Telnyx Edge Compute, in under 350 lines of TypeScript.

What You'll Build

An SMS conversation exporter that:

• Ingests SMS messages via Telnyx Messaging webhooks into a per-actor SQL database
• Counts and chunks messages for export (configurable chunk size, default 500)
• Uploads each chunk as a separate JSON file to Cloud Storage
• Writes a manifest file listing all chunks with metadata
• Sends an SMS notification when the export is complete (zero-credential binding)
• Handles 10,000+ messages via non-blocking, self-requeuing pipeline stages

The entire system runs on Telnyx Edge Compute — no external database, no separate upload service, no cross-cloud latency. The SQL database, the Cloud Storage upload, and the SMS notification all happen inside the same network boundary.

Why This Architecture Matters

If you've ever tried to export SMS conversation history for compliance or analytics, you've probably wired together:

• A webhook receiver (Lambda function or small server)
• A database (PostgreSQL, DynamoDB, or a hosted option)
• A batch export job (cron + a script that queries and writes to S3)
• A notification service (SNS, PagerDuty, or email)
• A separate orchestrator for chunking large datasets

That's five services, five deploys, five billing lines, and cross-region latency at every hop. The webhook arrives in one region, the database is in another, S3 is a third API call, and the notification is a fourth.

On Edge Compute with the Agent SDK, this collapses to one deploy. The webhook handler, the SQL database, the Cloud Storage upload, and the SMS sender are all bindings on the same actor — running at the carrier edge, in the same network boundary as the Messaging platform that sends the webhook.

Architecture

                    POST /export
                         │
                         ▼
              ┌────────────────────┐
              │   index.ts         │
              │   (HTTP router)    │
              └────┬───────────────┘
                   │
                   ▼
         ┌──────────────────────┐
         │  ExportAgent         │  (one actor per export job)
         │                      │
         │  1. countMessages()  │──► SQL DB: SELECT COUNT(*)
         │  2. exportChunk()    │──► SQL DB: SELECT chunk
         │                      │──► Cloud Storage: PUT JSON chunk
         │     (re-queues       │    (repeats until all chunks done)
         │      until done)     │
         │  3. writeManifest()  │──► Cloud Storage: PUT manifest.json
         │  4. notifyComplete() │──► SMS via [telnyx] binding
         └──────────────────────┘
                   │
                   ▼
         ┌──────────────────────┐
         │  Cloud Storage       │
         │  exports/{id}/       │
         │  ├── chunk-0000.json │
         │  ├── chunk-0001.json │
         │  └── manifest.json   │
         └──────────────────────┘

Quick Start

Prerequisites

• Node.js 18+
• Telnyx CLI (npm i -g @telnyx/cli)
• A Telnyx account with an API key, a messaging-enabled number, and a Cloud Storage bucket

1. Clone and install

git clone https://github.com/team-telnyx/telnyx-code-examples.git
cd telnyx-code-examples/sms-conversation-exporter
npm install

2. Configure

Edit telnyx.toml:

[storage.kv.EXPORT_KV]
id = "<your-kv-namespace-uuid>"

[storage.cloudstorage.EXPORT_STORAGE]
bucket_name = "<your-storage-bucket-name>"
region = "us-central-1"

[env_vars]
ALERT_PHONE = "+18005559876"
SENDER_PHONE = "+18005551234"
CHUNK_SIZE = "500"

3. Deploy

telnyx-edge secret set TELNYX_API_KEY KEY0123456789ABCDEF
telnyx-edge ship

4. Test the export pipeline

# Start an export (all conversations)
curl -X POST https://your-deployment.telnyxcompute.com/export

# Check progress
curl https://your-deployment.telnyxcompute.com/export/{exportId}

# Simulate 10,000 messages
curl -X POST https://your-deployment.telnyxcompute.com/simulate-bulk \
  -H "Content-Type: application/json" \
  -d '{"count": 10000}'

# Start a bulk export
curl -X POST https://your-deployment.telnyxcompute.com/export

How the Pipeline Works

Agent SDK queue() — Non-Blocking Stages

Each export job gets its own ExportAgent actor instance. The pipeline uses this.queue() to chain stages without blocking the HTTP request:

async start(params: { exportId: string; conversationFilter: string | null }): Promise<void> {
  await this.setState({ exportId: params.exportId, status: "counting", startedAt: Date.now() });
  await this.queue("countMessages");
}

Stage 1: Count

async countMessages(): Promise<void> {
  const result = this.ctx.storage.sql.exec("SELECT COUNT(*) as cnt FROM messages").toArray();
  const totalMessages = result[0].cnt;
  const totalChunks = Math.ceil(totalMessages / chunkSize);
  await this.setState({ totalMessages, totalChunks, status: "exporting" });
  await this.queue("exportChunk");
}

Stage 2: Export Chunk (Self-Requeuing)

async exportChunk(): Promise<void> {
  const state = await this.getState();
  const offset = state.chunkIndex * chunkSize;
  
  const rows = this.ctx.storage.sql.exec(
    "SELECT * FROM messages ORDER BY timestamp ASC LIMIT ? OFFSET ?",
    chunkSize, offset
  ).toArray();

  const chunkKey = `exports/${state.exportId}/chunk-${String(state.chunkIndex).padStart(4, "0")}.json`;
  await this.env.EXPORT_STORAGE.put(chunkKey, JSON.stringify(chunkData), {
    contentType: "application/json",
  });

  if (state.chunkIndex + 1 < state.totalChunks) {
    await this.queue("exportChunk");  // re-queue for next chunk
  } else {
    await this.queue("writeManifest");
  }
}

Stage 3: Write Manifest

async writeManifest(): Promise<void> {
  const manifest = {
    exportId: state.exportId,
    totalMessages: state.totalMessages,
    totalChunks: state.totalChunks,
    chunks: state.uploadedChunks,
  };
  await this.env.EXPORT_STORAGE.put(
    `exports/${state.exportId}/manifest.json`,
    JSON.stringify(manifest)
  );
  await this.queue("notifyComplete");
}

Stage 4: Notify (Zero-Credential SMS)

async notifyComplete(): Promise<void> {
  await this.env.TELNYX.messages.send({
    from: this.env.SENDER_PHONE,
    to: this.env.ALERT_PHONE,
    text: `Export complete: ${state.exportedMessages} messages in ${state.uploadedChunks.length} chunk(s).`,
  });
  await this.setState({ status: "done", completedAt: Date.now() });
}

The this.env.TELNYX.messages.send() call uses the zero-credential [telnyx] binding — no API key in code. The binding is declared in telnyx.toml and authenticated by the Edge Compute platform.

Chunked Output

Each chunk is a separate JSON file:

exports/export-1234567890-abc123/
├── chunk-0000.json    (messages 0–499)
├── chunk-0001.json    (messages 500–999)
├── ...
└── manifest.json      (metadata: total count, chunk list)

Each chunk contains:

{
  "exportId": "export-1234567890-abc123",
  "chunkIndex": 0,
  "totalChunks": 20,
  "totalMessages": 10000,
  "chunkSize": 500,
  "messages": [...],
  "exportedAt": 1718928001000
}

API Endpoints

Use Cases

• Compliance archival — Export SMS conversation history for regulatory compliance
• Data migration — Move SMS data from Edge SQL to a data warehouse via Cloud Storage
• Backup — Periodic JSON exports of all conversations to Cloud Storage
• Analytics — Export conversation data for offline analysis in Spark, BigQuery, or similar