Signalbus is a cloud-native, event-driven notification hub built with Go, Kafka, Docker, and cloud APIs (Mailgun, Twilio). It handles real-time email and SMS notifications through scalable microservices. Based on various events the data is put kafka through which the microservices get the data. Here microservices send emails and SMS. Its a simulation and a learning project.
/publish/:topicand/notifyendpoints- Kafka producer keyed by user; workers for email and SMS
- Email worker with retry + DLQ; Prometheus metrics and JSON logs
- SMS worker with retry + DLQ + per-tenant rate limit; Prometheus metrics
- Idempotency keys (DB unique index + cache)
- API key authentication; short- and long-term rate limiting
- PostgreSQL schema for notifications, attempts, keys, templates; search and redrive endpoints
- Templates with Go templates and localization
- Policy-driven fan-out honoring preferences
- Observability: standardized metrics, Grafana dashboards, OpenTelemetry tracing, alerting rules
- Multi-tenancy: tenant column propagation, scoped queries, quotas
Client Services (e.g., Order, Auth)
|
v
[ Notification API Service ]
|
v
Kafka Topics
|
v
[Email Worker] \[SMS Worker] \[Future Workers...]
|
v
Mailgun & Twilio
| Service | Description |
|---|---|
| Notification API | Accepts events and publishes to Kafka |
| Email Worker | Listens to Kafka and sends emails |
| SMS Worker | Listens to Kafka and sends SMS |
| Kafka/Zookeeper | Message broker infrastructure |
| Prometheus | Metrics collector |
| Grafana | Dashboard visualization |
- Go - Core microservices and libraries (
pkg/gomailer,pkg/gosms) - Kafka (segmentio/kafka-go) - Event distribution with consumer groups
- Docker / Docker Compose - Containerization and local orchestration
- Mailgun API + SMTP - Email delivery with fallback support
- Twilio API - SMS delivery with E.164 normalization
- PostgreSQL - Persistence for notifications, attempts, templates, tenants
- Redis - Caching and idempotency key storage
- Prometheus + Grafana - Metrics collection and visualization
- OpenTelemetry - Distributed tracing across services
- Railway + GitHub Actions - Cloud deployment and CI/CD pipeline
git clone https://github.com/jsndz/signalbus.git
cd signalbusRename .env.exampleto .env and replace the variables
docker-compose up --buildTest with Postman or curl:
curl -X POST http://localhost:8080/notify/signup \
-H "Content-Type: application/json" \
-d '{"email": "user@example.com", "phone": "+11234567890"}'How to Run Kafka separately:
- Run the docker image of
Kafkaandzoo-keeper(image details in docker-compose.yml)
Run docker compose(recommended):
- cd notification_hub
- docker compose build
- docker compose up -d
See logs:
- docker compose logs -f api
- docker compose logs -f email
- docker compose logs -f sms
Running API and Worker Separately (Manual Build):
-
docker build -f deployments/Dockerfile.api -t notification_hub:latest .
-
docker run -p 8080:8080 notification_hub:latest
-
docker build -f deployments/Dockerfile.email -t email:latest .
-
docker run -p 3001:3001 email:latest
-
docker build -f deployments/Dockerfile.sms -t sms:latest .
-
docker run -p 3000:3000 sms:latest
Stop everything:
- docker compose down
Example curl request:
curl -X POST http://localhost:8080/api/notify/signup \
-H "Content-Type: application/json" \
-d '{"username":"jsn", "password":"qwerty","email": "user@example.com", "phone": "+11234567890" }'
Response Example:
{
"message": "Signup event sent to Kafka"
}Logs:
email-1 | 2025/07/10 09:54:26 jsn qwerty
sms-1 | 2025/07/11 14:52:38 SMS sent to +91XXXXXXXXXX! Message SID: SMfXXXXXXXXXXXXXXXXXXXXXXXXXXX
email-1 | 202Topics Learned:
- Docker compose
- Twilio and Sendgrid Integration
- Kafka Producer and Consumer
- Kafka Retry Logics
- makefile
- Grafana + prometheus
- logging with zap
- Visit Prometheus:
http://localhost:9090 - Visit Grafana:
http://localhost:3002(default login:admin/admin)
- Services are deployed on Railway
- GitHub Actions handle CI/CD on push to
main - Kafka managed via Railway Add-on or Upstash
-
pkg/gomailer: interface + SMTP + Mailgun drivers; timeouts/retries; unit tests - Email worker: consumes
notifications.email, retry + DLQ, metrics, JSON logs -
pkg/gosms: interface; Twilio/Vonage drivers; E.164 normalization - SMS worker: retry + DLQ + per-tenant rate limit; metrics
- API:
/publish/:topic,/notify; API key auth; short/long-term rate limits - Idempotency keys (DB unique index + cache)
- OpenAPI spec + Swagger UI
- Kafka producer keyed by user (idempotent)
- Postgres: schemas for notifications, attempts, keys, templates
- Search +
GET /notifications/:id;POST /notifications/:id/redrive - Templates (Go tmpl) + localization; policy engine
- Observability: standardized metrics, Grafana dashboards, OTel tracing, alerting rules
- Multi-tenant: tenant column, scoped queries, per-tenant quotas
Pull requests are welcome! Open an issue first for feedback or discussion.
Current Phase: Advanced production-ready features (Phases 6-7 complete)
Key Libraries Built:
pkg/gomailer- Reusable email library with SMTP and Mailgun driverspkg/gosms- Reusable SMS library with Twilio driver and E.164 normalizationpkg/kafka- Kafka producer/consumer abstractionspkg/templates- Template engine with localization support
Production Patterns Implemented:
- Idempotency keys for duplicate prevention
- Exponential backoff retry with jitter
- Dead Letter Queue (DLQ) for failed messages
- Multi-tenant rate limiting and quotas
- Comprehensive observability with metrics, tracing, and logging
- API key authentication and authorization
- Template-based content rendering
V3 of Signalbus:
I am planning to remove the multi-tenancy and all SaaS level features from the project. Because the project is turning into mess. So I want build a highly scalable notification system.
- [] rate limiter for system after testing