SpaceLens Module 1 - Twitter Space Ingestion API

Module 1 of the SpaceLens (PayAI x402) project. This API accepts Twitter Space links, extracts metadata and media streams, and initializes the SpaceLens processing pipeline.

Features

• Accept Twitter Space URLs (live or recorded)
• Extract Space metadata using Twitter API v2
• Download and process Space recordings
• Extract audio from video files using FFmpeg
• Upload audio to Supabase Storage
• Queue jobs for transcription processing
• Track job progress and status

Tech Stack

• Node.js + Express
• Supabase (Postgres + Storage)
• BullMQ (Redis) for background job queue
• Twitter API v2 for Space data
• FFmpeg for audio extraction

Prerequisites

• Node.js v18 or higher
• Redis server (local or cloud)
• Supabase account and project
• Twitter API v2 Bearer Token (optional, uses mock data if not provided)

Installation

npm install

Configuration

Copy the .env file and configure the following variables:

# Supabase Configuration
VITE_SUPABASE_URL=your_supabase_url
VITE_SUPABASE_ANON_KEY=your_supabase_anon_key
SUPABASE_SERVICE_KEY=your_service_role_key

# Twitter API
TWITTER_BEARER_TOKEN=your_twitter_bearer_token

# Redis
REDIS_URL=redis://localhost:6379

# Server
PORT=3000

Database Setup

The database schema is automatically created through Supabase migrations:

• spaces table: Stores Twitter Space metadata
• jobs table: Tracks processing jobs
• space-audio storage bucket: Stores extracted audio files

Running the Application

Start the API Server

npm start

The server will start on http://localhost:3000

Start the Background Worker

In a separate terminal:

npm run worker

The worker listens to the transcription queue and processes jobs.

API Endpoints

POST /api/ingest

Ingest a Twitter Space URL and start processing.

Request Body:

{
  "twitter_space_url": "https://twitter.com/i/spaces/1YNGa..."
}

Response:

{
  "success": true,
  "spaceId": "1yNGaabcdEF",
  "jobId": "e3b9c03c-6e91-4bb2-b0da-fbaf16cbf7a7",
  "status": "ingested"
}

GET /api/space/:id

Get space metadata and job status.

Response:

{
  "success": true,
  "data": {
    "space": {
      "id": "uuid",
      "spaceId": "1yNGaabcdEF",
      "title": "Space Title",
      "type": "audio",
      "status": "ingested"
    },
    "jobs": [
      {
        "id": "uuid",
        "stage": "transcription",
        "progress": 10,
        "resultUrl": "https://..."
      }
    ]
  }
}

GET /health

Health check endpoint.

Project Structure

src/
├── config/          # Configuration management
├── routes/          # API route handlers
├── services/        # Business logic services
│   ├── database.js       # Supabase database operations
│   ├── twitterApi.js     # Twitter API integration
│   ├── mediaProcessor.js # FFmpeg audio processing
│   ├── queue.js          # BullMQ job queue
│   └── supabaseClient.js # Supabase client
├── utils/           # Helper functions
├── workers/         # Background job workers
└── server.js        # Express server entry point

Development Notes

• If Twitter API credentials are not provided, the system uses mock data for development
• Audio files are stored in the /tmp/spacelens directory during processing
• The worker process runs concurrently with the API server
• Jobs are automatically retried up to 3 times on failure

Next Steps

This module queues jobs for Module 2, which will handle:

• Audio transcription
• Content analysis
• AI-powered insights

License

Private - SpaceLens Project Ltd