PLAN.md

QueueKit – Implementation Plan

Phase 0 – Repo & Skeleton ✅

• Initialize Go module: github.com/reckziegelwilliam/queuekit
• Create basic structure:
  • cmd/queuekitd – server/worker binary
  • cmd/queuekit – CLI tool
  • internal/queue – core domain types
  • internal/backend – Postgres/Redis adapters
  • internal/worker – worker pool
  • internal/httpapi – HTTP handlers
  • internal/dashboard – templates and handlers
• Add Makefile / taskfile for common commands
• Set up Go linters and CI (GitHub Actions)

Phase 1 – Core Domain & Storage ✅

• Define job model:
  • Job (id, type, queue, payload, status, attempts, scheduled_at, etc.)
  • Queue model and statuses
• Implement backend interfaces:
  • Backend interface (enqueue, reserve, ack, nack, moveToDLQ, listQueues, listJobs)
  • Postgres implementation (including migrations)
  • Redis implementation (fast queue operations, locks)
• Unit tests for backend behavior

Phase 2 – Worker Pool & Execution ✅

• Implement worker pool:
  • Multiple workers per queue (configurable concurrency per queue)
  • Graceful shutdown (context cancellation, waits for in-flight jobs)
  • Worker status snapshots (idle / running / stopped + last job ID)
• Implement retry & backoff strategies:
  • Fixed backoff (FixedBackoff)
  • Exponential backoff (ExponentialBackoff, capped at MaxDelay)
  • Max attempts → DLQ (handled in backend Nack)
• Worker handler registration:
  • Map job type → handler func (Registry)
  • Context with job metadata passed to every handler
• Basic logging and instrumentation hooks (log/slog, per-event structured logs)
• Fix retry bug: backends now re-schedule non-final failures as pending with backoff delay

Phase 3 – HTTP API ✅

• Choose router (chi/v5)
• Implement endpoints:
  • POST /v1/jobs – enqueue
  • GET /v1/queues – list queues
  • GET /v1/queues/{name}/jobs – list jobs
  • GET /v1/jobs/{id} – get job
  • POST /v1/jobs/{id}/retry
  • POST /v1/jobs/{id}/cancel
  • DELETE /v1/jobs/{id} – delete job
• API key authentication (Bearer token middleware)
• JSON validation (leverages job.Validate())
• Integration tests (19 tests with mock backend)

Phase 4 – Dashboard ✅

• Setup html/template with embedded layout
• Views:
  • Queues overview (counts, health scores)
  • Queue details (jobs, status filter tabs, pagination)
  • Job details panel (payload, error, retry button)
• Minimal CSS (no framework, embedded in layout)
• Server-side sorting/filtering (no SPA)

Phase 5 – CLI (queuekit) ✅

• Set up CLI using cobra
• Commands:
  • queuekit enqueue <type> --queue <name> --payload <json>
  • queuekit inspect queues
  • queuekit inspect jobs --queue <name>
  • queuekit retry <job-id>
  • queuekit cancel <job-id>
• Global config via YAML (~/.config/queuekit/config.yaml) with viper

Phase 6 – Packaging & Examples ✅

• Multi-stage Dockerfile for queuekitd (distroless final image)
• docker-compose.yml with Postgres + Redis
• Example integrations:
  • Simple Go service (examples/simple/)
  • Cron-style recurring jobs (examples/cron/)
• README polish with API reference, CLI usage, architecture diagram