kumpul2

kumpul2 is a lightweight, configurable ETL (Extract, Transform, Load) service written in Go. It executes SQL queries against multiple configured PostgreSQL databases, writes the results to target tables, and optionally dumps them to JSON files. It supports both one-off execution and cron-based scheduling, with built-in Slack alerting.

Features

• Multi-Database Support: Execute queries across multiple configured PostgreSQL connections (pgx).
• Flexible Configuration: Split configuration for runtime options (schedule, alerts, DBs) and query definitions.
• Scheduling: Built-in cron scheduler or run-once mode.
• Alerting: Slack notifications with configurable severity thresholds (error, warning, info).
• Output Options:
  • Write results directly to target tables.
  • Dump results to JSON files for debugging or external processing.
• Safety & Observability:
  • Dry-run mode to validate configuration.
  • Structured logging via log/slog.
  • Timeout management for queries.

Prerequisites

• Go 1.25 or higher
• Docker (required only for running integration tests)

Configuration

Configuration is managed via TOML files.

1. Options (options.toml)

Defines the environment, database connections, and scheduling.

• cron: Cron expression for scheduling (e.g., "*/5 * * * *").
• slack_alert_url: Webhook URL for Slack notifications.
• slack_alert_level: Alert threshold (error, warning, info).
• timeout_seconds: Global timeout for query execution.
• [[db]]: Array of database configurations (id, connection string).

2. Queries (queries.toml)

Defines the ETL tasks.

• [[query]]: Array of query definitions.
  • id: Unique identifier for the query.
  • query: The SQL statement to execute.
  • target_table: The table where results should be written.

See config.example.toml and queries.example.toml for templates.

Usage

Run as a Service (Cron)

The default mode runs the scheduler continuously.

go run ./cmd/etl --option options.toml --query queries.toml

One-Off Execution

Run immediately and exit. Useful for manual runs or external schedulers.

go run ./cmd/etl --option options.toml --query queries.toml --once

Command Line Flags

Flag	Description
--option	Path to the options configuration file (default: options.toml).
--query	Path to the queries configuration file (default: queries.toml).
--verbose	Enable debug logging.
--dry-run	Validate configuration and connections without executing queries.
--dump	Save query results to JSON files.
--once	Execute all queries once and exit.

Development

Building

To build the project, run:

go build -o bin/etl ./cmd/etl

The binary will be located in the bin/ directory.

Running Tests

Unit tests:

go test ./...

Integration tests (requires Docker):

go test ./internal/integration/...