Durable Execution

A minimalistic durable task queue backed by PostgreSQL.

Conceptually based on Absurd, but implemented as an HTTP server instead of an embedded library, with additional features like a CLI and queue monitoring. The implementation is my own, with help from Claude.

Note that this is experimental software. For production workloads, please use an actual message queue such as SQS instead.

Features

• Durable tasks with retries — tasks survive process crashes. Failed attempts are retried automatically with configurable fixed or exponential backoff.
• Checkpointing — persist intermediate state keyed by (task, step). Retried tasks resume from the last checkpoint instead of starting over.
• Event coordination — runs can sleep until a named event arrives, enabling cross-workflow synchronization without polling.
• Long polling — claim requests hold the connection open until work is available, using PostgreSQL LISTEN/NOTIFY for near-instant wake-up.
• Workflow tracking — group related tasks under a workflow run for end-to-end observability.
• Queue monitoring — per-queue stats (pending, claimed, completed runs and consumer lag) for alerting and dashboards.
• Claim timeouts — if a worker doesn't complete or fail a run before the claim expires, the run becomes claimable again. No stuck tasks.
• Scheduling — defer runs to a future time.

Architecture

All state lives in PostgreSQL. There is no separate message broker — the database is the queue.

• Claiming uses SELECT ... FOR UPDATE SKIP LOCKED for contention-free, exactly-once delivery.
• Notifications use a trigger on the runs table that fires pg_notify with the queue name, waking only the relevant long-polling consumers.
• Transactions ensure that state transitions (claim, complete, fail, sleep, wake) are atomic.

API

The server exposes a REST API on port 8080. Interactive documentation is available at /docs (Swagger UI).

See api/openapi.yaml for the full specification.

Quick start

docker compose up

This starts PostgreSQL (with migrations applied automatically) and the server on port 8080.

# Create and claim a task
curl -s localhost:8080/api/v1/queues/demo/tasks \
  -d '{"task_name":"hello","params":{"msg":"world"}}'

curl -s localhost:8080/api/v1/queues/demo/tasks/claim \
  -d '{"limit":1,"claim_timeout":60}'

CLI

A command-line tool for interacting with the server.

go install ./cmd/durablectl

Make sure $GOPATH/bin is in your PATH: export PATH="$PATH:$(go env GOPATH)/bin"

Commands

durablectl queues create --name <queue>              Create a queue
durablectl queues list                               List all queues
durablectl queues delete <queue>                     Delete a queue
durablectl queues stats <queue>                      Get queue statistics
durablectl tasks create --queue <q> --name <task>    Create a task
durablectl tasks claim --queue <q>                   Claim tasks
durablectl tasks list                                List tasks
durablectl runs list --task <task_id>                List runs for a task
durablectl runs complete <run_id>                    Complete a run
durablectl runs fail <run_id> --error <msg>          Fail a run
durablectl events emit --name <event>                Emit an event

Example: queue stats

$ durablectl queues stats demo
{
  "queue_name": "demo",
  "pending_runs": 12,
  "claimed_runs": 3,
  "completed_runs": 47,
  "oldest_pending_run_age_seconds": 4.82
}

Use --server or DURABLE_SERVER to point at a different host:

durablectl --server https://prod:8080 queues stats demo

Clients

• Go — clients/go
• Python — clients/python (sync, based on httpx)
• CLI — cmd/durablectl

Project structure

api/                  OpenAPI spec and Go API types
clients/go/           Go HTTP client
clients/python/       Python HTTP client
cmd/durablectl/       CLI tool
examples/             Example workflows
internal/
  db/                 Store, listener, migrations
  db/gen/             sqlc-generated code
  db/queries/         SQL queries
  server/             HTTP handlers and routing
  service/            Business logic