ymlink is a webhook-driven workflow engine that listens for events and executes predefined workflows asynchronously. Workflows are defined in a YAML file and can be used to orchestrate tasks like calling external APIs, sending notifications, and more.
- Webhook Trigger: Start workflows by sending a POST request to an endpoint.
- Asynchronous Execution: Events are processed in the background by a worker pool.
- YAML-based Workflows: Define complex workflows with steps, conditions, and loops in a simple YAML format.
- Dynamic Payloads: Use Go templates to dynamically construct HTTP requests based on event data.
- Authentication: Secure your webhook endpoint with bearer token or HMAC signature validation.
- Retry Logic: Configure automatic retries for failed HTTP requests with exponential backoff.
- Graceful Shutdown: Ensures that in-flight jobs are completed before the application shuts down.
- Go 1.18 or higher
-
Clone the repository:
git clone https://github.com/your-username/ymlink.git cd ymlink -
Install dependencies:
go mod tidy
ymlink is configured using a YAML file (e.g., ymlink.yaml).
# ymlink - Main Configuration
# Server settings
server:
# The address the server will listen on.
addr: ":8080"
# Path to the workflow definition file.
workflows_path: "workflows.yaml"
# Authentication settings to secure the /events endpoint.
auth:
# Bearer token authentication.
# The client must send an "Authorization: Bearer <token>" header.
# To disable, leave this empty.
bearer: "a-very-secret-token"
# HMAC signature validation (optional, more secure).
# If set, the client must send an "X-Signature" header with the HMAC-SHA256
# of the request body, and a "X-Timestamp" header.
# hmac_secret: "your-hmac-secret-key"
# Maximum allowed time difference (in seconds) for HMAC timestamp validation.
max_skew_seconds: 300
# A list of allowed IP addresses or CIDR ranges.
# If empty, all IPs are allowed.
allowlist:
- "127.0.0.1/32"
- "192.168.1.0/24"
# A collection of named service endpoints that can be referenced in workflows.
# This allows you to manage URLs in a central location.
endpoints:
# Example API for testing HTTP requests
httpbin_api: "https://httpbin.org"
# Example Slack incoming webhook URL
slack_webhook_url: "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
# A collection of secret values (API keys, tokens, etc.)
# These are securely stored and can be referenced in workflows.
secrets:
# Example API key for a service
some_api_key: "key-for-some-service"
# Slack bot token (if needed for more complex interactions)
slack_bot_token: "xoxb-your-slack-bot-token"addr: The address and port the server listens on.workflows_path: The path to your workflow definitions file.
bearer: A secret token for bearer authentication. Clients must include this token in theAuthorizationheader.hmac_secret: A secret key for HMAC-SHA256 signature validation. This provides a higher level of security.max_skew_seconds: The allowed time difference for HMAC timestamp validation.allowlist: A list of IP addresses or CIDR ranges that are allowed to access the/eventsendpoint.
A map of named URLs that can be used in your workflows. This helps to avoid hardcoding URLs in your workflow definitions.
A map of secret values, such as API keys or tokens. These are securely stored and can be accessed in your workflows.
Run the application:
make runThis will build and start the ymlink server.
Workflows are defined in a YAML file (e.g., workflows.yaml) and consist of a list of named workflows. Each workflow has a series of steps.
This workflow is triggered by an event named user_created. It first sends a notification to Slack and then, if successful, logs the event data to another service using an API key for authentication.
- name: "user_created"
steps:
# Step 1: Send a welcome message to a Slack channel.
- name: "Send Welcome Message to Slack"
id: "send_to_slack"
url: "{{ .Endpoints.slack_webhook_url }}"
body_template: |
{
"text": "New user created: {{ .Event.Data.username }} ({{ .Event.Data.email }})"
}
headers:
Content-Type: "application/json"
retry:
max_attempts: 3
base_delay_ms: 100
max_delay_ms: 1000
# Step 2: Log the full event data to an external service using a bearer token.
# This step demonstrates using a secret as a bearer token for authentication.
- name: "Log User Data with Auth"
if: "{{ .Results.send_to_slack }}" # This step only runs if the previous step was successful
url: "{{ .Endpoints.httpbin_api }}/post"
body_template: '{{ .Event.Data | to_json }}'
headers:
Content-Type: "application/json"
Authorization: "Bearer {{ .Secrets.some_api_key }}"This workflow, triggered by a process_items event, demonstrates how to iterate over a list of items sent in the event payload. For each item, it makes a separate HTTP request.
To trigger this workflow, you would send a POST request to /events with a body like this:
{
"name": "process_items",
"data": {
"items": [
{ "name": "Item A", "value": 123 },
{ "name": "Item B", "value": 456 }
]
}
}- name: "process_items"
steps:
- name: "Iterate Over Items"
type: "for_each"
# The 'items' field must render to a valid JSON array.
# Here, we get the array from the incoming event data.
items: '{{ .Event.Data.items | to_json }}'
# The 'step_template' is executed for each item in the array.
# The current item is available as the '.Item' variable.
step_template:
name: "Process Item: {{ .Item.name }}"
url: "{{ .Endpoints.httpbin_api }}/post"
body_template: |
{
"item_name": "{{ .Item.name }}",
"item_value": "{{ .Item.value }}",
"correlation_id": "{{ .Event.CorrelationID }}"
}
headers:
Content-Type: "application/json"POST /events: The main endpoint for triggering workflows. The request body should be a JSON object with annamefield corresponding to a workflow name.GET /healthz: A health check endpoint that returns200 OK.GET /version: Returns the application version.
To build the application, run:
make buildThis will create a binary named ymlink in the project root.
This project is licensed under the MIT License. See the LICENSE file for details.