📖 README: Collaborative Data Grid Backend

🎯 Project Goal

This project implements the backend for a real-time, horizontally scalable collaborative data grid (simulating a modern spreadsheet application). It is built to demonstrate how Redis can be used in high-performance TypeScript backends, and implement operational scaling.

---

🏗️ Technical Stack

Component	Technology	Rationale / Key Feature
Backend	TypeScript (Node.js, Express)	Ensures type-safety and robust application logic.
Real-Time Layer	WebSockets (ws)	Allows for persistent, low-latency, two-way communication between the client and server.
Database	PostgreSQL	Used as the single source of truth for structured grid and cell data.
ORM	Prisma	Provides a type-safe interface for database interactions and manages connection pooling effectively.
Scaling Bridge	Redis Pub/Sub	The critical mechanism enabling horizontal scaling of the Node.js application.

---

🌐 Architecture and Scaling Rationale (The Value Proposition)

The core challenge of a collaborative application is ensuring real-time updates across multiple server instances (horizontal scaling).

1. The Real-Time Loop (Horizontal Scaling)

We use Redis Pub/Sub as a central, high-speed message bus to decouple senders from receivers:

• Server Instances: All backend instances are subscribed to Redis channels (e.g., grid:<id>).
• Update Flow: When a client sends a cell update via WebSocket to Server A:
  1. Persist: Server A writes the change to PostgreSQL (via Prisma).
  2. Publish: Server A uses its Redis Publisher to send the update to the specific channel (grid:<id>).
  3. Broadcast: All other servers (e.g., Server B) instantly receive the message via their Redis Subscriber and broadcast the update to their locally connected clients via WebSockets.

This architecture ensures linear scalability: we can launch multiple Node.js instances to handle massive WebSocket traffic, and collaboration remains seamless across all users.

2. Best Practices & Performance

• Prisma Singleton: The PrismaClient is implemented using the recommended Singleton Pattern (globalThis), which prevents connection pool exhaustion in the database, crucial for handling concurrent read/write operations efficiently.
• Decoupling: The WebSocket Server is explicitly attached to the underlying Node.js HTTP server, allowing a single port to efficiently handle both REST API traffic and persistent WebSocket connections.

---

💻 Setup and Development

Prerequisites

• Docker and Docker Compose (CLI required on the host).
• Node.js and npm (required on the host for running npm install for the Zed Language Server).
• Development Environment: Configured to use a remote Docker host via SSH.

Quick Start

1. Configure Secrets: Create a .env file in the project root to define your credentials:

   # .env (copy from .env.example)
   DB_USER=mysecureuser
   DB_PASSWORD=myverysupersecretpassword123
   DB_NAME=causaldb
   DB_PORT=5432
   REDIS_HOST=redis
   REDIS_PORT=6379
   

2. Launch Stack: This command reads the .env file, builds the image, and launches the three services (app, db, cache):

   docker-compose up --build

3. Initialize DB Schema: Run the Prisma migration (requires the stack to be running):

   npx prisma migrate dev --name init_grid_schema

4. Install Host Dependencies: Run npm install on the remote host to satisfy the Zed/TypeScript language server (not the Docker container):

   npm install

---

🧪 Testing the Solution

The backend listens for HTTP and WebSocket traffic on the mapped host port (e.g., 8081).

1. Create a Grid (REST Test):

   # Use your mapped port, e.g., 8081
   curl -X POST http://localhost:8081/api/grids -H "Content-Type: application/json" -d '{"name": "ExampleGrid", "rows": 5, "cols": 5}'

2. Verify Real-Time (End-to-End):
   • Open two browser tabs and navigate to http://<YOUR_SERVER_IP>:8081/.
   • The single-page client will connect via WebSocket to ws://.../test-grid-1.
   • Type a value in any cell in Browser 1.
   • Confirm the value updates instantly in Browser 2, verifying the Postgres write $\to$ Redis publish $\to$ WebSocket broadcast loop.

---

Demo