perf: research real-time event notification alternatives to reduce polling cost

[lancy]

Background

The CLI currently uses short polling (500ms interval) to fetch agent run events from the server. While this approach is simple and serverless-friendly, it may incur unnecessary server costs due to frequent requests.

Current Implementation

• Location: turbo/apps/cli/src/commands/run.ts:185-263
• Polling interval: 500ms (2 requests/second)
• Endpoint: GET /api/agent/runs/{runId}/events
• Data sources: Events from Axiom, status from PostgreSQL

CLI (Polling Client)
    |
    | HTTP GET (every 500ms)
    v
Vercel Serverless Function
    |
    +---> PostgreSQL (run status)
    +---> Axiom (agent events)
    |
    v
JSON Response

Research Findings

SSE (Server-Sent Events) - Not Recommended

Vercel officially states:

"SSE is an inefficient transport. It keeps a persistent connection open between client and server, even when idle."

Problems with SSE on Vercel:

• Persistent connections incur continuous billing (entire connection duration)
• Function duration limits (300s-800s max) would require reconnection logic
• SSE endpoint would still need to poll Axiom internally
• MCP specification deprecated SSE in favor of Streamable HTTP (March 2025)

Long Polling - Mixed Results

• Fewer requests but longer execution time per request
• Still holds connection open, consuming serverless execution time
• Net cost benefit unclear

Vercel's Recommended Approaches

1. Third-party Pub/Sub services (Pusher, Ably, PubNub, etc.)

   • Client subscribes directly to the service
   • Serverless function only publishes events (quick execution)
   • Real-time push without holding connections

2. SWR (Stale-While-Revalidate)

   • Vercel's own dashboard uses this pattern
   • On-demand fetching with intelligent caching
   • Supports adaptive polling intervals

Cost Comparison (Theoretical)

Approach	Vercel Cost	Third-party Cost
Short Polling (current)	~100ms execution × 2/sec	None
SSE	Entire connection duration	None
Pusher/Ably	Minimal (publish only)	~$49/month minimum

Proposal

Consider integrating Pusher (or similar service like Ably) for real-time event delivery:

How It Would Work

Sandbox                      Vercel                     Pusher                CLI
   |                           |                          |                    |
   |-- webhook (events) ------>|                          |                    |
   |                           |-- trigger(event) ------->|                    |
   |                           |                          |-- push ----------->|
   |                           |                          |                    |

Benefits

• Reduced Vercel execution time (only publish, no polling)
• Lower latency (real-time push vs 500ms polling)
• Better scalability (Pusher handles connections)
• Vercel officially recommends this pattern

Trade-offs

• Additional service cost (~$49+/month)
• New dependency to manage
• Requires CLI to include Pusher client SDK

Research Tasks

• Evaluate Pusher vs Ably vs alternatives for our use case
• Calculate actual cost savings vs third-party service fees
• Prototype Pusher integration
• Measure latency improvement
• Consider fallback strategy (polling as backup)

References

• Vercel: Publish and Subscribe to Realtime Data
• Vercel: Deploying Pusher Channels
• Vercel: Building Efficient MCP Servers
• Why MCP Deprecated SSE
• Ably vs Pusher Comparison

---

🤖 Generated from conversation with Claude Code

[e7h4n]

Innovation: Ably Presence-Based Direct Assignment

Core Concept

Eliminate thundering herd by pre-assigning tasks to specific runners.

Instead of broadcasting notifications to all runners (causing a race), the API proactively selects the best runner and sends the task directly to that runner's private channel.

Requirements Addressed

• ✅ 1K-1M user-level isolation: Each user has dedicated channels
• ✅ Non-idempotent tasks: Tasks assigned once, never retried on failure
• ✅ Long-running tasks: Heartbeat mechanism via presence updates
• ✅ 10-100 concurrent tasks per runner: API tracks real-time capacity
• ✅ Public internet: Ably accessible from anywhere
• ✅ Managed service: No infrastructure to manage
• ✅ Zero thundering herd: Each task sent to exactly one runner

Architecture

1. Runners Register via Ably Presence
   ↓
   Runner A joins presence channel:
   {
     runnerId: "runner-a",
     userId: 123,
     capacity: 10,      // Max concurrent tasks
     available: 8,      // Current available slots
     lastSeen: timestamp
   }

2. API Queries Live Runners
   ↓
   Task created for user 123
   → API calls: GET /channels/runner-presence:user-123/presence
   → Response: [runner-a (8 slots), runner-b (10 slots), runner-c (5 slots)]

3. API Selects Best Runner
   ↓
   Choose runner-b (most available capacity: 10 slots)
   → UPDATE agent_runs SET status='assigned', assigned_runner='runner-b'

4. API Sends to Runner's Private Channel
   ↓
   Publish to: runner-tasks:user-123:runner-b
   Message: {
     runId: "run_xyz",
     executionContext: { prompt, agentComposeId, sandboxToken, ... }
   }

5. Runner Executes (No Claim Needed)
   ↓
   Runner B receives message
   → Update presence: available: 9
   → Execute task
   → Heartbeat every 30s via presence update
   → Complete: POST /api/webhooks/agent/complete
   → Update presence: available: 10

Key Components

1. Runner Presence Registration

// Runner startup
const presenceChannel = ably.channels.get('runner-presence:user-123');

// Join presence set
await presenceChannel.presence.enter({
  runnerId: 'runner-a',
  userId: 123,
  capacity: 10,
  available: 10,
  version: '1.0.0',
  startedAt: Date.now()
});

// Heartbeat loop (every 30s)
setInterval(async () => {
  await presenceChannel.presence.update({
    available: getCurrentAvailableSlots(),
    lastSeen: Date.now()
  });
}, 30000);

// Subscribe to private task channel
const taskChannel = ably.channels.get('runner-tasks:user-123:runner-a');
taskChannel.subscribe('task', async (message) => {
  // Direct execution - no claim API needed
  await executeTask(message.data);
});

2. API Task Assignment

async function createRun(userId, prompt, ...) {
  // 1. Insert into PostgreSQL
  const run = await db.insert(agentRuns).values({
    userId,
    status: 'pending',
    prompt,
    ...
  });

  // 2. Query online runners from Ably Presence
  const response = await ably.request('GET',
    `/channels/runner-presence:user-${userId}/presence`
  );
  const runners = response.items;

  if (runners.length === 0) {
    // No online runners - task stays pending
    // Runners will pull on startup
    return run;
  }

  // 3. Select best runner (most available capacity)
  const bestRunner = runners
    .sort((a, b) => b.data.available - a.data.available)[0];

  // 4. Assign in database
  await db.update(agentRuns)
    .set({
      status: 'assigned',
      assignedRunner: bestRunner.clientId
    })
    .where(eq(agentRuns.id, run.id));

  // 5. Send to runner's private channel
  const channel = ably.channels.get(
    `runner-tasks:user-${userId}:${bestRunner.clientId}`
  );

  await channel.publish('task', {
    runId: run.id,
    executionContext: {
      prompt,
      agentComposeId,
      sandboxToken,
      ...
    }
  });

  return run;
}

3. Runner Crash Recovery

// API subscribes to presence leave events
const presenceChannel = ably.channels.get('runner-presence:user-123');

presenceChannel.presence.subscribe('leave', async (member) => {
  console.log(`Runner ${member.clientId} crashed/disconnected`);

  // Query orphaned tasks
  const orphanedRuns = await db.select()
    .from(agentRuns)
    .where(
      and(
        eq(agentRuns.assignedRunner, member.clientId),
        or(
          eq(agentRuns.status, 'assigned'),
          eq(agentRuns.status, 'running')
        )
      )
    );

  // Reassign to another runner
  for (const run of orphanedRuns) {
    await reassignRun(run.id, run.userId);
  }
});

Crash Detection: Ably presence heartbeat (every 15s). If 30s without heartbeat → presence leave event.

4. Cold Start (No Runners Online)

// Runner startup pulls pending tasks
async function onRunnerStart(userId, runnerId) {
  // 1. Join presence
  await presenceChannel.presence.enter({...});

  // 2. Pull pending tasks
  const response = await fetch('/api/runners/pending', {
    method: 'POST',
    body: JSON.stringify({
      userId,
      runnerId,
      maxTasks: 10  // Batch pull
    })
  });

  const tasks = await response.json();

  // 3. Execute
  for (const task of tasks) {
    executeTask(task);
  }
}

API endpoint for batch claiming:

-- Atomic batch assignment
BEGIN;

SELECT run_id, prompt, ...
FROM agent_runs
WHERE user_id = $1
  AND status = 'pending'
ORDER BY created_at
LIMIT $2
FOR UPDATE SKIP LOCKED;

UPDATE agent_runs
SET status = 'assigned',
    assigned_runner = $3
WHERE run_id IN (...);

COMMIT;

Authentication & Isolation

Runner Token (Ably Capabilities)

// API endpoint: POST /api/runners/ably-token
async function issueAblyToken(runnerId, userId) {
  // Verify runner belongs to user
  const runner = await verifyRunner(runnerId, userId);

  // Generate Ably capability token
  const token = await ably.auth.createTokenRequest({
    clientId: runnerId,
    capability: {
      // Presence channel (join, update)
      [`runner-presence:user-${userId}`]: ['presence', 'subscribe'],
      // Private task channel (subscribe only)
      [`runner-tasks:user-${userId}:${runnerId}`]: ['subscribe']
    },
    ttl: 3600000  // 1 hour
  });

  return token;
}

Security enforcement:

• Runner can only join presence for their own user
• Runner can only subscribe to their own private task channel
• API verifies runner ownership before issuing token

User Isolation

Complete isolation at every level:

1. Presence channels: runner-presence:user-{userId} - separate per user
2. Task channels: runner-tasks:user-{userId}:{runnerId} - private per runner
3. PostgreSQL: WHERE user_id = {userId} enforced in all queries
4. Ably capabilities: Token restricts access to user's channels only

Advantages

✅ Zero thundering herd: Tasks sent to exactly one runner
✅ No wasted API calls: No claim race, no failed attempts
✅ Smart load balancing: API sees real-time capacity of all runners
✅ Lower latency: ~30ms (no claim API round-trip)
✅ Perfect isolation: User-level channel separation
✅ Crash recovery: Presence leave event (15-30s detection)
✅ Managed service: Ably handles all infrastructure
✅ Scales to millions: Ably designed for massive scale
✅ Batch efficiency: Runners can claim multiple tasks on startup

Cost Analysis

For 10K active users:

• Base: $49/month
• Connections: 10K runners × $0.001 = $10/month
• Presence updates (heartbeat): 10K × 2/min × 43,200 min/month × $0.04/M = $34/month
• Task messages: 100K/day × 30 days × $0.04/M = $3.6/month

Total: ~$97/month for 10K users

For 100K active users:

• Base: $49/month
• Connections: 100K × $0.001 = $100/month
• Presence updates: 100K × 2/min × 43,200 min/month × $0.04/M = $345/month
• Task messages: 1M/day × 30 days × $0.04/M = $36/month

Total: ~$530/month for 100K users

Scaling behavior: Linear with number of active runners.

Comparison to Alternatives

Feature	Broadcast + Claim	Ably Presence (This)
Thundering herd	⚠️ All runners race	✅ Zero (directed)
Wasted API calls	⚠️ Many failures	✅ None
Load balancing	❌ Random	✅ Capacity-aware
Latency	~50ms (notify+claim)	~30ms (direct)
Crash recovery	PostgreSQL cleanup (60s)	Presence leave (15-30s)
Complexity	Low	Medium (presence mgmt)
Cost (10K users)	~$150/mo	~$97/mo

Trade-offs

Pros:

• Best performance (lowest latency, zero waste)
• Intelligent scheduling (capacity-aware)
• Fast crash detection

Cons:

• Higher implementation complexity (presence management)
• Requires API to query Ably presence on every task creation
• Presence leave detection has 15-30s delay

Implementation Checklist

• Set up Ably account and configure project
• Implement runner presence registration
• Implement API presence query + selection logic
• Create private channel naming scheme
• Implement token issuance endpoint
• Add presence leave event handler for crash recovery
• Implement cold start batch claiming
• Add monitoring for presence set size
• Test runner crash scenarios
• Load test with 100+ concurrent runners per user

Next Steps

1. Validate requirements: Confirm this approach meets all needs
2. Prototype: Build MVP with 2-3 runners to test presence mechanism
3. Load test: Test with target scale (100 runners per user)
4. Move to /deep-plan: Create detailed implementation plan

---

This approach completely eliminates the thundering herd problem while providing intelligent load balancing and fast crash recovery.