Deep Dive: Web Dashboard (Express/Fastify + Playwright)

[chhot2u]

Web Dashboard (Express/Fastify + Playwright) — Deep Dive Analysis

Overview

Server-side architecture with a web-based dashboard. No desktop app needed — access from any browser. Playwright workers run headless on the server. Top-ranked solution (8.40/10).

---

Architecture

┌─────────────────────────────────────────────────┐
│              Client (Any Browser)                │
│  ┌───────────────────────────────────────────┐  │
│  │   React/Vue/Svelte SPA                    │  │
│  │   - Real-time task dashboard (WebSocket)  │  │
│  │   - Live screenshot stream                │  │
│  │   - Proxy configuration panel             │  │
│  │   - Auth credential vault                 │  │
│  │   - Results explorer + export             │  │
│  └───────────────────────────────────────────┘  │
├─────────────────────────────────────────────────┤
│              Server (Node.js)                    │
│  ┌───────────────────────────────────────────┐  │
│  │   Express/Fastify API                     │  │
│  │   - REST endpoints + WebSocket            │  │
│  │   - Auth middleware (JWT)                  │  │
│  │   - Rate limiting                         │  │
│  ├───────────────────────────────────────────┤  │
│  │   BullMQ Task Queue                       │  │
│  │   - Priority-based scheduling             │  │
│  │   - Retry with exponential backoff        │  │
│  │   - Concurrency: 100 workers              │  │
│  │   - Dead letter queue                     │  │
│  ├───────────────────────────────────────────┤  │
│  │   Playwright Worker Pool                  │  │
│  │   - 1 Browser, 100 BrowserContexts       │  │
│  │   - Per-context proxy + cookies           │  │
│  │   - Screenshot streaming via WebSocket    │  │
│  │   - Network interception & logging        │  │
│  ├───────────────────────────────────────────┤  │
│  │   Data Layer                              │  │
│  │   - PostgreSQL (task state, results)      │  │
│  │   - Redis (queue, cache, pub/sub)         │  │
│  │   - S3/MinIO (screenshots, artifacts)     │  │
│  └───────────────────────────────────────────┘  │
└─────────────────────────────────────────────────┘

---

Key Dependencies

{
  "fastify": "^5.x",
  "@fastify/websocket": "^11.x",
  "playwright-core": "^1.49.x",
  "bullmq": "^5.x",
  "ioredis": "^5.x",
  "pg": "^8.x",
  "jsonwebtoken": "^9.x",
  "zod": "^3.x"
}

---

Proxy per Task Implementation

// Worker processor
import { Worker } from 'bullmq';
import { chromium, BrowserContext } from 'playwright-core';

let browser: Browser;

async function initBrowser() {
  browser = await chromium.launch({ headless: true });
}

const worker = new Worker('tasks', async (job) => {
  const { proxy, steps, taskId } = job.data;
  
  const context = await browser.newContext({
    proxy: {
      server: proxy.server,
      username: proxy.username,
      password: proxy.password,
    },
    locale: proxy.locale,
    timezoneId: proxy.timezone,
    userAgent: proxy.userAgent,
    viewport: { width: 1920, height: 1080 },
  });

  const page = await context.newPage();

  // Stream screenshots to dashboard
  page.on('framenavigated', async () => {
    const screenshot = await page.screenshot({ type: 'jpeg', quality: 50 });
    pubsub.publish(`task:${taskId}:screenshot`, screenshot);
  });

  try {
    for (const step of steps) {
      await executeStep(page, step);
      await job.updateProgress(step.index / steps.length * 100);
    }
    return { success: true, results: await collectResults(page) };
  } finally {
    await context.close();
  }
}, { concurrency: 100 });

---

Scaling Beyond 100

// Horizontal scaling with multiple worker processes
import cluster from 'node:cluster';
import os from 'node:os';

if (cluster.isPrimary) {
  const WORKERS = Math.min(os.cpus().length, 4);
  for (let i = 0; i < WORKERS; i++) {
    cluster.fork(); // Each worker handles 25 contexts
  }
} else {
  // Each worker creates its own browser + 25 contexts
  startWorker({ concurrency: 25 });
}

---

Strengths

• No desktop app overhead: Zero bundle, access from anywhere
• Horizontally scalable: Add servers for >100 tasks
• BullMQ reliability: Persistent queue, retries, dead letter queue
• Real-time dashboard: WebSocket for live updates + screenshots
• Team accessible: Multiple users can monitor from any browser
• Easy deployment: Docker, Kubernetes, any cloud provider
• CI/CD friendly: Integrate with existing pipelines
• Familiar stack: Pure Node.js, no exotic runtimes

Weaknesses

• No offline use: Requires running server
• Server costs: Need to host and maintain
• Not a desktop app: No native OS integration (tray, notifications)
• Redis dependency: Required for BullMQ queue
• Network latency: Dashboard updates depend on connection

---

Resource Estimates (100 tasks)

Resource	Estimate
RAM	~2 GB (headless contexts are lighter)
CPU	4 cores minimum, 8 recommended
Disk	~100MB app + Playwright browsers
Redis	~50MB
PostgreSQL	Depends on data volume
Startup	<1s server, ~2s browser

---

Deployment Options

# docker-compose.yml
services:
  api:
    build: ./api
    ports: ["3000:3000"]
    depends_on: [redis, postgres]
  
  worker:
    build: ./worker
    deploy:
      replicas: 4  # 4 workers x 25 contexts = 100 tasks
    depends_on: [redis, postgres]
  
  dashboard:
    build: ./dashboard
    ports: ["8080:80"]
  
  redis:
    image: redis:7-alpine
  
  postgres:
    image: postgres:16-alpine

---

When to Choose This Stack

✅ Need to scale beyond 100 tasks
✅ Multiple users need dashboard access
✅ Want easy deployment (Docker/K8s)
✅ Already have server infrastructure
✅ Need CI/CD integration
✅ Team knows Node.js/TypeScript

❌ Avoid if: must be offline desktop app, need native OS features, or zero infrastructure

---

Verdict: 8.40/10 — TOP RECOMMENDATION

Best balance of scalability, developer experience, and maintainability. Easiest to scale from 100 to 1000+ tasks.

References issue #1 for full comparison