DAGSmith — Workflow Compiler + Parallel Execution Runner

A full-stack web application for visually designing workflows (nodes + edges), with a backend that validates, compiles, and executes workflows using a job queue with parallelism, retries, idempotency, and live streaming updates.

Features

• Visual Workflow Editor - React Flow-based drag-and-drop interface
• 8 Node Types - START, END, DELAY, TRANSFORM, HTTP_REQUEST, CONDITION, PARALLEL_SPLIT, MERGE
• DAG Compiler - Validates workflows, detects cycles, computes topological order and parallel levels
• Parallel Execution - BullMQ workers with concurrency control
• Live Updates - Socket.io streaming of node status and logs
• Retry & Idempotency - Configurable retry policies with exponential backoff
• Execution Timeline - Gantt-style visualization of node execution
• Critical Path Analysis - Post-execution metrics and parallelism gains

Tech Stack

Frontend

• React + TypeScript
• React Flow (workflow graph builder)
• Zustand (state management)
• Tailwind CSS
• Socket.io client
• React Query

Backend

• Node.js + Express
• Socket.io server
• BullMQ + Redis (job queue)
• PostgreSQL + Prisma (ORM)
• Zod (validation)

Quick Start

Prerequisites

• Node.js 18+
• Docker & Docker Compose
• pnpm (recommended) or npm

1. Start Infrastructure

# Start PostgreSQL and Redis
npm run docker:up

2. Install Dependencies

npm install

3. Setup Database

# Create .env file for API (copy from example)
cp apps/api/.env.example apps/api/.env

# Generate Prisma client and run migrations
cd apps/api
npx prisma generate
npx prisma migrate dev --name init

# Seed demo workflows
npm run db:seed

4. Start Development Servers

# From project root - starts both API and web
npm run dev

• Frontend: http://localhost:5173
• Backend API: http://localhost:4000
• API Health: http://localhost:4000/health

Project Structure

/apps
  /web          # React frontend
    /src
      /components
      /pages
      /stores
      /lib
  /api          # Express backend
    /src
      /routes
      /services
      /workers
      /lib
    /prisma

/packages
  /shared       # Shared types & schemas

/infra
  docker-compose.yml

API Endpoints

Workflows

• POST /api/workflows - Create workflow
• GET /api/workflows - List workflows
• GET /api/workflows/:id - Get workflow
• POST /api/workflows/:id/versions - Save new version
• POST /api/workflows/:id/validate - Validate workflow
• POST /api/workflows/:id/run - Start execution

Executions

• GET /api/executions/:id - Get execution details
• GET /api/executions/:id/node-runs - Get node run statuses
• GET /api/executions/:id/logs - Get execution logs
• POST /api/executions/:id/nodes/:nodeId/retry - Retry failed node
• POST /api/executions/:id/cancel - Cancel execution
• POST /api/executions/:id/rerun - Re-run execution

Node Types

Type	Description	Config
START	Entry point	initialContext
END	Exit point	—
DELAY	Wait timer	delayMs
TRANSFORM	JS expression	expression
HTTP_REQUEST	HTTP call	method, url, headers, bodyTemplate, timeoutMs
CONDITION	Branch logic	predicate (returns true/false)
PARALLEL_SPLIT	Fan-out	—
MERGE	Fan-in	strategy (object/array)

WebSocket Events

Connect to Socket.io and join execution rooms for live updates:

socket.emit('join:execution', executionId);

socket.on('node:status', (event) => {
  // { executionId, nodeId, status, attempt, ts, durationMs, output, error }
});

socket.on('node:log', (event) => {
  // { executionId, nodeId, nodeRunId, ts, level, message, meta }
});

socket.on('execution:status', (event) => {
  // { executionId, status, ts, summary }
});

Demo Workflows

Three pre-built templates are seeded:

1. HTTP + Transform + Condition - Fetches API data, transforms, and branches
2. Parallel Fan-out - 3 parallel HTTP requests that merge
3. Retry Showcase - Demonstrates retry behavior

Environment Variables

API (apps/api/.env)

DATABASE_URL="postgresql://dagsmith:dagsmith_secret@localhost:5432/dagsmith"
REDIS_URL="redis://localhost:6379"
PORT=4000
NODE_ENV=development

Development

# Run both frontend and backend
npm run dev

# Run only backend
npm run dev:api

# Run only frontend
npm run dev:web

# Database commands
npm run db:generate   # Regenerate Prisma client
npm run db:migrate    # Run migrations
npm run db:seed       # Seed demo data

# Docker
npm run docker:up     # Start Postgres + Redis
npm run docker:down   # Stop containers

Architecture Highlights

Compiler

• Validates DAG structure (cycles, reachability, node rules)
• Computes topological order
• Calculates parallel execution levels
• Validates node configs against Zod schemas

Execution Engine

• BullMQ worker with global concurrency (10)
• Per-execution concurrency tracking via Redis
• Idempotency via unique constraint on (executionId, nodeId, attempt)
• Exponential backoff retries
• Critical path calculation post-execution

Real-time Updates

• Socket.io rooms per execution
• Events emitted on every status change
• Frontend updates node colors + logs live

License

MIT