diff --git a/README.md b/README.md index 5b031a7..091194c 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,8 @@ Refer to the [**Quick Start Guide**](docs/QUICKSTART.md) for detailed onboarding - ✅ **Open source** (Apache 2.0) - ✅ **Privacy-by-design** (Immutable consent ledger, orchestrated erasure) -- ✅ **Cloud-native** (Kubernetes-ready, Kafka event-driven, stateless) +- ✅ **Cloud-native** (Kubernetes-ready, OpenTelemetry-native, stateless) +- ✅ **Infrastructure Agnostic** (Support for PostgreSQL/MySQL, Kafka/RabbitMQ) - ✅ **High Performance** (gRPC aggregation at the gateway layer) - ✅ **Compliant** with GDPR, ePrivacy, and emerging data laws @@ -71,17 +72,18 @@ PCM uses **Hexagonal Architecture** with clear **bounded contexts**, standardize | Service | Responsibility | Stack | |---------|----------------|-------| -| **Profile Service** | User identity, handle management, dynamic attributes | PostgreSQL (JSONB), Redis | -| **Consent Service** | GDPR consent collection, versioning, legal proof | PostgreSQL (Ledger) | -| **Segment Service** | User classification and real-time segmentation | Elasticsearch, Kafka | +| **Profile Service** | User identity, handle management, dynamic attributes | PostgreSQL/MySQL (JSON), Redis | +| **Consent Service** | GDPR consent collection, versioning, legal proof | PostgreSQL/MySQL (Ledger) | +| **Segment Service** | User classification and real-time segmentation | Elasticsearch, Kafka/RabbitMQ | | **Preference Service** | UX preferences (language, theme, notifications) | Redis | -| **Config Service** | Centralized configuration for the entire platform | Spring Cloud Config | -| **API Gateway** | Unified entry point, JWT security, Aggregator | Spring Cloud Gateway, gRPC | +| **Config Service** | (Optional) Centralized configuration | Spring Cloud Config | +| **API Gateway** | Unified entry point, JWT security, Aggregator | Gateway, gRPC, OTel | ### Communication - **Synchronous**: gRPC for high-performance data aggregation (Gateway -> Services) -- **Asynchronous**: Kafka with Avro schemas for cross-service events (e.g., GDPR Erasure) +- **Asynchronous**: Spring Cloud Stream with support for **Kafka** (Avro) and **RabbitMQ** +- **Observability**: **OpenTelemetry** native (TRACES & METRICS exporters) - **Security**: JWT for client authentication, standard OAuth2 resources server --- @@ -109,7 +111,11 @@ pcm/ --- ## ⚙️ Configuration -PCM uses a centralized configuration model via **Spring Cloud Config**. All core platform settings (Kafka, Redis, Vault, Logging) are managed in the `config-service`. +PCM uses a dual configuration model: +1. **Centralized**: Via **Spring Cloud Config** (recommended for production). +2. **Decentralized**: All services provide local fallbacks and can be configured entirely via **Environment Variables** (ideal for local dev or simple container deployments). + +Core platform settings (Messaging, Database, Vault, Logging) are standardized in the `config-service`. - **Source of truth**: `config-service/src/main/resources/config/` - **Shared config**: `application.yml` @@ -119,7 +125,8 @@ PCM uses a centralized configuration model via **Spring Cloud Config**. All core ## 📚 Documentation - [**Quick Start Guide**](docs/QUICKSTART.md) - Get PCM running locally in 5 minutes. -- [**API Reference**](docs/API_REFERENCE.md) - Endpoints, payloads, and Examples. +- [**API Reference**](docs/API_REFERENCE.md) - Endpoints, payloads, and examples. +- [**Dependency Monitoring**](docs/DEPENDENCY_MONITORING.md) - Guide to monitoring updates and vulnerabilities. - [**Architecture Decision Records**](docs/architecture/) - Design decisions and rationale. --- diff --git a/api-gateway/pom.xml b/api-gateway/pom.xml index 2dd318c..bb5f060 100644 --- a/api-gateway/pom.xml +++ b/api-gateway/pom.xml @@ -45,6 +45,14 @@ io.micrometer micrometer-registry-prometheus + + io.micrometer + micrometer-tracing-bridge-otel + + + io.opentelemetry + opentelemetry-exporter-otlp + @@ -62,7 +70,7 @@ net.devh grpc-client-spring-boot-starter - 2.15.0.RELEASE + ${grpc-spring-boot-starter.version} io.grpc @@ -79,7 +87,7 @@ net.logstash.logback logstash-logback-encoder - 7.4 + ${logstash-logback-encoder.version} diff --git a/api-gateway/src/main/resources/application.yml b/api-gateway/src/main/resources/application.yml index f563b19..bcf631f 100644 --- a/api-gateway/src/main/resources/application.yml +++ b/api-gateway/src/main/resources/application.yml @@ -2,4 +2,17 @@ spring: application: name: api-gateway config: - import: "optional:configserver:http://localhost:8888" + import: "optional:configserver:${CONFIG_SERVER_URL:http://localhost:8888}" + cloud: + config: + fail-fast: false + gateway: + routes: + - id: profile-service + uri: ${PROFILE_SERVICE_URL:http://localhost:18081} + predicates: + - Path=/api/v1/profiles/** + - id: consent-service + uri: ${CONSENT_SERVICE_URL:http://localhost:18083} + predicates: + - Path=/api/v1/consents/** diff --git a/config-service/src/main/resources/config/api-gateway.yml b/config-service/src/main/resources/config/api-gateway.yml index 7bc56ee..48c7be7 100644 --- a/config-service/src/main/resources/config/api-gateway.yml +++ b/config-service/src/main/resources/config/api-gateway.yml @@ -30,7 +30,7 @@ spring: oauth2: resourceserver: jwt: - issuer-uri: ${JWT_ISSUER_URI:http://localhost:8090/auth/realms/pcm} + issuer-uri: ${JWT_ISSUER_URI:http://localhost:8090/realms/pcm} # Keycloak URL without /auth resilience4j: circuitbreaker: diff --git a/config-service/src/main/resources/config/application.yml b/config-service/src/main/resources/config/application.yml index 1a3ff48..1eb7548 100644 --- a/config-service/src/main/resources/config/application.yml +++ b/config-service/src/main/resources/config/application.yml @@ -15,7 +15,7 @@ spring: show-sql: false properties: hibernate: - dialect: org.hibernate.dialect.PostgreSQLDialect + dialect: ${DB_DIALECT:org.hibernate.dialect.PostgreSQLDialect} format_sql: true jdbc: time_zone: UTC @@ -43,6 +43,12 @@ spring: configuration: schema.registry.url: ${SCHEMA_REGISTRY_URL:http://localhost:8081} specific.avro.reader: true + rabbit: + binder: + host: ${RABBITMQ_HOST:localhost} + port: ${RABBITMQ_PORT:5672} + username: ${RABBITMQ_USERNAME:guest} + password: ${RABBITMQ_PASSWORD:guest} bindings: # Producers (profile-service) profileCreated-out-0: @@ -87,6 +93,13 @@ spring: enabled: true backend: secret + # Security Abstraction + security: + pii: + provider: ${PII_PROTECTION_PROVIDER:vault} # Options: vault, local + local: + secret: ${PII_LOCAL_SECRET:abcdefghijklmnop} # 16 chars for AES-128 + # Shared Management & Observability management: endpoints: @@ -105,6 +118,16 @@ management: tracing: sampling: probability: 1.0 + otlp: + tracing: + endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4318/v1/traces} + otlp: + metrics: + export: + url: ${OTEL_EXPORTER_OTLP_METRICS_ENDPOINT:http://localhost:4318/v1/metrics} + enabled: true + tracing: + endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4318/v1/traces} # Shared Logging logging: diff --git a/consent-service/pom.xml b/consent-service/pom.xml index 608e8c3..11bcdbc 100644 --- a/consent-service/pom.xml +++ b/consent-service/pom.xml @@ -58,7 +58,7 @@ net.devh grpc-server-spring-boot-starter - 2.15.0.RELEASE + ${grpc-spring-boot-starter.version} @@ -66,6 +66,10 @@ org.springframework.cloud spring-cloud-starter-stream-kafka + + org.springframework.cloud + spring-cloud-stream-binder-rabbit + @@ -97,6 +101,10 @@ io.micrometer micrometer-tracing-bridge-otel + + io.opentelemetry + opentelemetry-exporter-otlp + @@ -116,7 +124,7 @@ net.logstash.logback logstash-logback-encoder - 7.4 + ${logstash-logback-encoder.version} diff --git a/consent-service/src/main/java/dev/vibeafrika/pcm/consent/domain/model/Consent.java b/consent-service/src/main/java/dev/vibeafrika/pcm/consent/domain/model/Consent.java index d9120b5..2262be1 100644 --- a/consent-service/src/main/java/dev/vibeafrika/pcm/consent/domain/model/Consent.java +++ b/consent-service/src/main/java/dev/vibeafrika/pcm/consent/domain/model/Consent.java @@ -5,6 +5,8 @@ import lombok.AccessLevel; import lombok.Getter; import lombok.NoArgsConstructor; +import org.hibernate.annotations.JdbcTypeCode; +import org.hibernate.type.SqlTypes; import org.springframework.data.annotation.CreatedDate; import org.springframework.data.jpa.domain.support.AuditingEntityListener; @@ -18,8 +20,8 @@ */ @Entity @Table(name = "consent_ledger", indexes = { - @Index(name = "idx_consent_profile", columnList = "profile_id, timestamp DESC"), - @Index(name = "idx_consent_purpose", columnList = "profile_id, purpose, timestamp DESC") + @Index(name = "idx_consent_profile", columnList = "profile_id, timestamp DESC"), + @Index(name = "idx_consent_purpose", columnList = "profile_id, purpose, timestamp DESC") }) @EntityListeners(AuditingEntityListener.class) @Getter @@ -61,12 +63,13 @@ public class Consent implements AggregateRoot { @Column(name = "proof_hash", nullable = false, length = 64) private String proofHash; - @Column(columnDefinition = "JSONB") + @JdbcTypeCode(SqlTypes.JSON) + @Column(name = "metadata") private String metadata; - private Consent(String tenantId, UUID profileId, ConsentPurpose purpose, boolean granted, - String version, String consentText, String ipAddress, - String userAgent, String proofHash) { + private Consent(String tenantId, UUID profileId, ConsentPurpose purpose, boolean granted, + String version, String consentText, String ipAddress, + String userAgent, String proofHash) { this.id = UUID.randomUUID(); this.tenantId = tenantId; this.profileId = profileId; @@ -79,15 +82,15 @@ private Consent(String tenantId, UUID profileId, ConsentPurpose purpose, boolean this.proofHash = proofHash; } - public static Consent grant(String tenantId, UUID profileId, ConsentPurpose purpose, String version, - String consentText, String ipAddress, String userAgent, - String proofHash) { + public static Consent grant(String tenantId, UUID profileId, ConsentPurpose purpose, String version, + String consentText, String ipAddress, String userAgent, + String proofHash) { return new Consent(tenantId, profileId, purpose, true, version, consentText, ipAddress, userAgent, proofHash); } - public static Consent revoke(String tenantId, UUID profileId, ConsentPurpose purpose, String version, - String consentText, String ipAddress, String userAgent, - String proofHash) { + public static Consent revoke(String tenantId, UUID profileId, ConsentPurpose purpose, String version, + String consentText, String ipAddress, String userAgent, + String proofHash) { return new Consent(tenantId, profileId, purpose, false, version, consentText, ipAddress, userAgent, proofHash); } } diff --git a/consent-service/src/main/resources/application.yml b/consent-service/src/main/resources/application.yml index 27fed93..832e22b 100644 --- a/consent-service/src/main/resources/application.yml +++ b/consent-service/src/main/resources/application.yml @@ -2,4 +2,19 @@ spring: application: name: consent-service config: - import: "optional:configserver:http://localhost:8888" + import: "optional:configserver:${CONFIG_SERVER_URL:http://localhost:8888}" + cloud: + config: + fail-fast: false + # Local fallback for Kafka if config-server is down + stream: + kafka: + binder: + brokers: ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092} + + # Local fallback for Datasource + datasource: + url: jdbc:postgresql://${DB_HOST:localhost}:${DB_PORT:8843}/consent_db + username: ${DB_USERNAME:pcm} + password: ${DB_PASSWORD:pcm_dev_password} + driver-class-name: org.postgresql.Driver diff --git a/consent-service/src/main/resources/db/migration/V1__create_consent_ledger.sql b/consent-service/src/main/resources/db/migration/V1__create_consent_ledger.sql index 469f110..6b8ac86 100644 --- a/consent-service/src/main/resources/db/migration/V1__create_consent_ledger.sql +++ b/consent-service/src/main/resources/db/migration/V1__create_consent_ledger.sql @@ -6,11 +6,11 @@ CREATE TABLE consent_ledger ( granted BOOLEAN NOT NULL, version VARCHAR(20) NOT NULL, consent_text TEXT NOT NULL, - timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(), + timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, ip_address VARCHAR(45), user_agent TEXT, proof_hash VARCHAR(64) NOT NULL, - metadata JSONB + metadata JSON ); CREATE INDEX idx_consent_profile ON consent_ledger(profile_id, timestamp DESC); diff --git a/docker-compose.yml b/docker-compose.yml index 4504cf0..c5220b8 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -253,16 +253,27 @@ services: image: jaegertracing/all-in-one:latest container_name: pcm-jaeger ports: - - "5775:5775/udp" - - "6831:6831/udp" - - "6832:6832/udp" - - "5778:5778" - - "16686:16686" - - "14268:14268" - - "14250:14250" - - "9411:9411" - environment: - COLLECTOR_ZIPKIN_HOST_PORT: :9411 + - "16686:16686" # UI + - "4317:4317" # OTLP gRPC (internal collector) + - "4318:4318" # OTLP HTTP (internal collector) + - "9411:9411" # Zipkin (legacy) + networks: + - pcm-network + + # OpenTelemetry Collector + otel-collector: + image: otel/opentelemetry-collector-contrib:latest + container_name: pcm-otel-collector + command: ["--config=/etc/otel-collector-config.yaml"] + volumes: + - ./docker/otel-collector-config.yaml:/etc/otel-collector-config.yaml + ports: + - "4317:4317" # OTLP gRPC + - "4318:4318" # OTLP HTTP + - "8888:8888" # Metrics + depends_on: + - jaeger + - prometheus networks: - pcm-network @@ -291,6 +302,24 @@ services: networks: - pcm-network + # RabbitMQ - Alternative messaging broker for portability verification + rabbitmq: + image: rabbitmq:3-management-alpine + container_name: pcm-rabbitmq + ports: + - "5672:5672" # AMQP + - "15672:15672" # Management UI + environment: + RABBITMQ_DEFAULT_USER: guest + RABBITMQ_DEFAULT_PASS: guest + healthcheck: + test: ["CMD", "rabbitmq-diagnostics", "check_running"] + interval: 10s + timeout: 5s + retries: 5 + networks: + - pcm-network + volumes: postgresql_data: redis_data: diff --git a/docker/otel-collector-config.yaml b/docker/otel-collector-config.yaml new file mode 100644 index 0000000..cde55ec --- /dev/null +++ b/docker/otel-collector-config.yaml @@ -0,0 +1,29 @@ +receivers: + otlp: + protocols: + grpc: + http: + +processors: + batch: + +exporters: + logging: + loglevel: debug + otlp/jaeger: + endpoint: jaeger:4317 + tls: + insecure: true + prometheus: + endpoint: "0.0.0.0:8888" + +service: + pipelines: + traces: + receivers: [otlp] + processors: [batch] + exporters: [logging, otlp/jaeger] + metrics: + receivers: [otlp] + processors: [batch] + exporters: [logging, prometheus] diff --git a/docs/DEPENDENCY_MONITORING.md b/docs/DEPENDENCY_MONITORING.md new file mode 100644 index 0000000..ee20a9e --- /dev/null +++ b/docs/DEPENDENCY_MONITORING.md @@ -0,0 +1,75 @@ +# Dependency Monitoring and Updates + +This document describes the tools and processes put in place to monitor the dependencies of the PCM project and ensure they remain secure and up to date. + +## 1. Local Tools (Maven) + +We have configured two essential Maven plugins in the root `pom.xml` to allow you to check the status of your dependencies locally. + +### Checking for new versions + +The `versions-maven-plugin` has been added. It allows you to see which libraries have newer versions available. + +**Command:** + +```bash +mvn versions:display-dependency-updates +``` + +*Note: This will check all modules of the project.* + +### Checking vulnerabilities (CVE) + +The `dependency-check-maven` (OWASP) plugin has been added. It scans your dependencies for known vulnerabilities (CVEs). + +**Command:** + +```bash +mvn org.owasp:dependency-check-maven:check +``` + +*Note: The report will be generated in `target/dependency-check-report.html`. Open this file in your browser to see the details.* + +> [!TIP] +> **NVD API Key**: The first run allows downloading the vulnerability database, which can fail (403/404) due to NVD rate limits. +> It is highly recommended to obtain an [NVD API Key](https://nvd.nist.gov/developers/request-an-api-key) and configure it in your `settings.xml` or via command line: +> `mvn org.owasp:dependency-check-maven:check -DnvdApiKey=YOUR_KEY` + +--- + +## 2. Automation (CI/CD) + +For continuous monitoring without manual intervention, we recommend using tools connected to your code repository (GitHub/GitLab). + +### GitHub Dependabot (Recommended) + +If your code is hosted on GitHub, Dependabot is the simplest solution. It automatically creates Pull Requests to update vulnerable or outdated dependencies. + +**Configuration:** +Create a `.github/dependabot.yml` file at the root of the project: + +```yaml +version: 2 +updates: + - package-ecosystem: "maven" + directory: "/" + schedule: + interval: "weekly" + groups: + spring-boot: + patterns: + - "org.springframework.boot:*" + - "org.springframework.cloud:*" +``` + +### Renovate Bot + +[Renovate](https://docs.renovatebot.com/) is a more configurable alternative to Dependabot, capable of grouping updates (“Grouped Updates”) to avoid Pull Request “noise”. + +--- + +## 3. Best Practices + +1. **Regular updates**: Run the `mvn versions:display-dependency-updates` command at the beginning of each sprint. +2. **Security overrides**: If a critical vulnerability is discovered in a transitive dependency (e.g., `netty`), use the `` section of the root `pom.xml` to force a secure version (as we did for the recent remediation). +3. **Automated tests**: Never merge a dependency update unless the test suite (`mvn test`) passes successfully. diff --git a/docs/PORTABILITY.md b/docs/PORTABILITY.md new file mode 100644 index 0000000..9960673 --- /dev/null +++ b/docs/PORTABILITY.md @@ -0,0 +1,59 @@ +# Infrastructure Portability & Configuration + +PCM is designed to be highly portable and infrastructure-agnostic. This document explains how to switch between different infrastructure providers using environment variables. + +## 1. Messaging (Kafka vs RabbitMQ) + +PCM uses Spring Cloud Stream to abstract the messaging layer. + +| Provider | Enabled via environment variables | +| :--- | :--- | +| **Kafka** (Default) | `SPRING_CLOUD_STREAM_BINDER=kafka`, `KAFKA_BOOTSTRAP_SERVERS` | +| **RabbitMQ** | `SPRING_CLOUD_STREAM_BINDER=rabbit`, `RABBITMQ_HOST`, `RABBITMQ_PORT`, `RABBITMQ_USERNAME`, `RABBITMQ_PASSWORD` | + +--- + +## 2. Database (PostgreSQL vs MySQL) + +The database layer has been generalized to support any major relational database. + +| Setting | Variable | Default | +| :--- | :--- | :--- | +| **URL** | `SPRING_DATASOURCE_URL` | `jdbc:postgresql://localhost:8843/pcm_db` | +| **Username** | `DB_USERNAME` | `pcm` | +| **Password** | `DB_PASSWORD` | `pcm_dev_password` | +| **Dialect** | `DB_DIALECT` | `org.hibernate.dialect.PostgreSQLDialect` | + +> [!TIP] +> To use MySQL, set `DB_DIALECT=org.hibernate.dialect.MySQLDialect` and update the connection URL. + +--- + +## 3. PII Protection (Vault vs Local) + +The Sensitive Data (PII) protection layer is abstracted. + +| Provider | Variable | Description | +| :--- | :--- | :--- | +| **Vault** (Default) | `PII_PROTECTION_PROVIDER=vault` | Uses HashiCorp Vault Transit Engine. | +| **Local** | `PII_PROTECTION_PROVIDER=local` | Uses localized AES-128 encryption with a static key (`PII_LOCAL_SECRET`). | + +--- + +## 4. Observability (OpenTelemetry) + +Tracing and Metrics are standardized on the OpenTelemetry (OTLP) protocol. + +| Setting | Variable | Default | +| :--- | :--- | :--- | +| **OTLP Endpoint** | `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318/v1/traces` | +| **Metrics Endpoint** | `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | `http://localhost:4318/v1/metrics` | + +--- + +## 5. Configuration (Centralized vs Decentralized) + +| Mode | Import String | Description | +| :--- | :--- | :--- | +| **Centralized** | `spring.config.import=configserver:http://localhost:8888` | Fetches config from Config Service. | +| **Decentralized** | `spring.config.import=optional:configserver:...` | Falls back to local/env properties if Config Server is down. | diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md index 300b567..fb8690e 100644 --- a/docs/QUICKSTART.md +++ b/docs/QUICKSTART.md @@ -33,12 +33,14 @@ docker-compose up -d ``` This will start: -- **PostgreSQL**: Profile and Consent storage (Port 8843) -- **Redis**: Caching and Preference storage (Port 6779) -- **Kafka & Zookeeper**: Event distribution -- **Elasticsearch**: Segmentation engine -- **Schema Registry**: Port 8081 -- **Kafka UI**: Accessible at `http://localhost:8095` +- **PostgreSQL/MySQL**: Storage (Port 8843) +- **Redis**: Caching (Port 6779) +- **Kafka**: Messaging (Port 9092) +- **RabbitMQ**: Alternative Messaging (Port 5672) +- **Elasticsearch**: Segmentation (Port 9200) +- **OpenTelemetry Collector**: Observability (Port 4318) +- **Jaeger**: Distributed Tracing (Port 16686) +- **Kafka UI**: `http://localhost:8095` ## 2. Build the Platform @@ -105,6 +107,8 @@ curl -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: vibe-afrika" http://loca | **Consent** | 18083 | `http://localhost:18083/swagger-ui.html` | | **Segment** | 18084 | `http://localhost:18084/swagger-ui.html` | | **Config** | 8888 | `http://localhost:8888/actuator/health` | +| **Tracing (Jaeger)** | 16686 | `http://localhost:16686/search` | +| **Rabbit UI** | 15672 | `http://localhost:15672` | --- **Note**: In development mode, security is relaxed for some endpoints. In production, all requests except profile registration require a valid JWT. diff --git a/docs/architecture/adr-003-infrastructure-abstraction.md b/docs/architecture/adr-003-infrastructure-abstraction.md new file mode 100644 index 0000000..830c99f --- /dev/null +++ b/docs/architecture/adr-003-infrastructure-abstraction.md @@ -0,0 +1,29 @@ +# ADR 003: Infrastructure Abstraction Layer + +## Status +Accepted + +## Context +PCM aims to be a portable, open-source identity platform. The initial implementation was tightly coupled to specific infrastructure providers: +- **PostgreSQL** (specifically `JSONB` and index types) +- **Kafka** (hard dependency on binder-specific logic) +- **Vault** (mandatory for PII encryption) +- **Spring Cloud Config** (mandatory for bootstrap) +- **Brave/Zipkin** (specific tracing headers) + +To increase adoption and deployment flexibility, we needed to abstract these layers. + +## Decision +Implement a "Generic Infrastructure" layer using Spring Boot's abstraction capabilities: +1. **Config**: Use `optional:configserver` and environment variable fallbacks for a decentralized mode. +2. **Database**: Standardize on Hibernate 6 native JSON mapping and generic SQL types in migrations to support MySQL/MariaDB/Postgres. +3. **Messaging**: Use Spring Cloud Stream Binders to allow swappable backends (Kafka/RabbitMQ). +4. **Encryption**: Implement a `PIIProtectionProvider` interface to support Vault or local AES encryption. +5. **Observability**: Migrate to OpenTelemetry (OTLP) for vendor-neutral tracing and metrics. + +## Consequences +- **Positive**: Simplified local development (no mandatory Config Server). +- **Positive**: Platform can be deployed on a wider range of Cloud managed services (RDS MySQL, Cloud PubSub, etc.). +- **Positive**: Future-proof observability using industry standards. +- **Negative**: Slight increase in dependency management overhead in the parent `pom.xml`. +- **Negative**: Need to maintain multiple binders/dialects if vendor-specific features are required. diff --git a/libs/common/pom.xml b/libs/common/pom.xml index 635df9d..095551b 100644 --- a/libs/common/pom.xml +++ b/libs/common/pom.xml @@ -19,6 +19,12 @@ Shared utilities, interfaces, and domain primitives + + + org.slf4j + slf4j-api + + jakarta.validation diff --git a/libs/common/src/main/java/dev/vibeafrika/pcm/common/security/LocalAesPiiProtectionProvider.java b/libs/common/src/main/java/dev/vibeafrika/pcm/common/security/LocalAesPiiProtectionProvider.java new file mode 100644 index 0000000..ebe2330 --- /dev/null +++ b/libs/common/src/main/java/dev/vibeafrika/pcm/common/security/LocalAesPiiProtectionProvider.java @@ -0,0 +1,59 @@ +package dev.vibeafrika.pcm.common.security; + +import lombok.extern.slf4j.Slf4j; + +import javax.crypto.Cipher; +import javax.crypto.spec.SecretKeySpec; +import java.nio.charset.StandardCharsets; +import java.util.Base64; + +/** + * A simple PII protection provider using local AES encryption. + * Primarily intended for development or small-scale deployments without Vault. + */ +@Slf4j +public class LocalAesPiiProtectionProvider implements PiiProtectionProvider { + + private final SecretKeySpec secretKey; + private static final String ALGORITHM = "AES"; + + public LocalAesPiiProtectionProvider(String secret) { + if (secret == null || secret.length() != 16) { + throw new IllegalArgumentException("Secret key must be 16 characters long for AES-128"); + } + this.secretKey = new SecretKeySpec(secret.getBytes(StandardCharsets.UTF_8), ALGORITHM); + } + + @Override + public String encrypt(String plainText) { + if (plainText == null || plainText.isEmpty()) { + return plainText; + } + try { + Cipher cipher = Cipher.getInstance(ALGORITHM); + cipher.init(Cipher.ENCRYPT_MODE, secretKey); + byte[] encrypted = cipher.doFinal(plainText.getBytes(StandardCharsets.UTF_8)); + return "local:v1:" + Base64.getEncoder().encodeToString(encrypted); + } catch (Exception e) { + log.error("Error encrypting PII locally", e); + throw new RuntimeException("Encryption failed", e); + } + } + + @Override + public String decrypt(String cipherText) { + if (cipherText == null || !cipherText.startsWith("local:v1:")) { + return cipherText; + } + try { + String base64Encrypted = cipherText.substring("local:v1:".length()); + Cipher cipher = Cipher.getInstance(ALGORITHM); + cipher.init(Cipher.DECRYPT_MODE, secretKey); + byte[] decrypted = cipher.doFinal(Base64.getDecoder().decode(base64Encrypted)); + return new String(decrypted, StandardCharsets.UTF_8); + } catch (Exception e) { + log.error("Error decrypting PII locally", e); + throw new RuntimeException("Decryption failed", e); + } + } +} diff --git a/libs/common/src/main/java/dev/vibeafrika/pcm/common/security/PiiProtectionProvider.java b/libs/common/src/main/java/dev/vibeafrika/pcm/common/security/PiiProtectionProvider.java new file mode 100644 index 0000000..ee49a95 --- /dev/null +++ b/libs/common/src/main/java/dev/vibeafrika/pcm/common/security/PiiProtectionProvider.java @@ -0,0 +1,25 @@ +package dev.vibeafrika.pcm.common.security; + +/** + * Interface for protecting Personally Identifiable Information (PII). + * This allows the platform to abstract the underlying encryption mechanism + * (Vault, Cloud KMS, Local AES, etc.). + */ +public interface PiiProtectionProvider { + + /** + * Encrypts a plaintext string. + * + * @param plainText the string to encrypt + * @return the encrypted ciphertext + */ + String encrypt(String plainText); + + /** + * Decrypts a ciphertext string. + * + * @param cipherText the ciphertext to decrypt + * @return the original plaintext + */ + String decrypt(String cipherText); +} diff --git a/libs/common/src/test/java/dev/vibeafrika/pcm/common/security/LocalAesPiiProtectionProviderTest.java b/libs/common/src/test/java/dev/vibeafrika/pcm/common/security/LocalAesPiiProtectionProviderTest.java new file mode 100644 index 0000000..53d4284 --- /dev/null +++ b/libs/common/src/test/java/dev/vibeafrika/pcm/common/security/LocalAesPiiProtectionProviderTest.java @@ -0,0 +1,46 @@ +package dev.vibeafrika.pcm.common.security; + +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class LocalAesPiiProtectionProviderTest { + + private final String VALID_SECRET = "1234567890123456"; // 16 chars + + @Test + void shouldEncryptAndDecryptSuccessfully() { + LocalAesPiiProtectionProvider provider = new LocalAesPiiProtectionProvider(VALID_SECRET); + String original = "sensitive data"; + + String encrypted = provider.encrypt(original); + assertThat(encrypted).startsWith("local:v1:"); + assertThat(encrypted).isNotEqualTo(original); + + String decrypted = provider.decrypt(encrypted); + assertThat(decrypted).isEqualTo(original); + } + + @Test + void shouldReturnSameValueIfNullOrEmpty() { + LocalAesPiiProtectionProvider provider = new LocalAesPiiProtectionProvider(VALID_SECRET); + + assertThat(provider.encrypt(null)).isNull(); + assertThat(provider.encrypt("")).isEmpty(); + assertThat(provider.decrypt(null)).isNull(); + } + + @Test + void shouldNotDecryptIfPrefixMissing() { + LocalAesPiiProtectionProvider provider = new LocalAesPiiProtectionProvider(VALID_SECRET); + String notEncrypted = "some-text"; + + assertThat(provider.decrypt(notEncrypted)).isEqualTo(notEncrypted); + } + + @Test + void shouldThrowExceptionIfSecretInvalid() { + assertThatThrownBy(() -> new LocalAesPiiProtectionProvider("too-short")) + .isInstanceOf(IllegalArgumentException.class); + } +} diff --git a/libs/kafka-events/pom.xml b/libs/kafka-events/pom.xml index 361cb2e..c789d96 100644 --- a/libs/kafka-events/pom.xml +++ b/libs/kafka-events/pom.xml @@ -36,6 +36,16 @@ io.confluent kafka-avro-serializer + + + org.apache.commons + commons-lang3 + + + + org.lz4 + lz4-java + diff --git a/pom.xml b/pom.xml index ad02213..c3337ef 100644 --- a/pom.xml +++ b/pom.xml @@ -31,34 +31,36 @@ UTF-8 - 3.2.1 - 2023.0.0 + 3.3.2 + 2023.0.3 42.7.7 - 10.4.1 - 6.4.1.Final + 10.11.1 + 6.4.4.Final - 3.9.1 + 3.7.1 1.11.4 7.6.1 - 8.11.3 + 8.13.4 3.2.1 - 1.77 + 1.78 1.60.1 4.28.2 + 3.1.0.RELEASE 1.12.1 1.34.1 + 7.4 5.10.1 @@ -141,6 +143,16 @@ kafka-avro-serializer ${confluent.version} + + org.springframework.cloud + spring-cloud-stream-binder-kafka + 4.1.0 + + + org.springframework.cloud + spring-cloud-stream-binder-rabbit + 4.1.0 + org.apache.avro avro @@ -190,6 +202,16 @@ pom import + + io.micrometer + micrometer-tracing-bridge-otel + 1.2.2 + + + io.opentelemetry + opentelemetry-exporter-otlp + ${opentelemetry.version} + @@ -212,6 +234,52 @@ ${rest-assured.version} test + + + + org.apache.commons + commons-lang3 + 3.14.0 + + + org.lz4 + lz4-java + 1.8.0 + + + + + org.assertj + assertj-core + 3.25.3 + + + org.apache.commons + commons-compress + 1.26.1 + + + commons-io + commons-io + 2.16.1 + + + io.netty + netty-bom + 4.1.111.Final + pom + import + + + ch.qos.logback + logback-classic + 1.5.6 + + + net.minidev + json-smart + 2.5.1 + @@ -333,7 +401,6 @@ - org.apache.avro avro-maven-plugin @@ -347,6 +414,25 @@ + + + + org.codehaus.mojo + versions-maven-plugin + 2.16.2 + + false + + + + org.owasp + dependency-check-maven + 9.0.9 + + 7 + false + + diff --git a/preference-service/pom.xml b/preference-service/pom.xml index 6f171bc..4aaa979 100644 --- a/preference-service/pom.xml +++ b/preference-service/pom.xml @@ -29,6 +29,18 @@ org.springframework.boot spring-boot-starter-actuator + + io.micrometer + micrometer-registry-prometheus + + + io.micrometer + micrometer-tracing-bridge-otel + + + io.opentelemetry + opentelemetry-exporter-otlp + org.springframework.cloud spring-cloud-starter-config @@ -45,6 +57,10 @@ org.springframework.cloud spring-cloud-starter-stream-kafka + + org.springframework.cloud + spring-cloud-stream-binder-rabbit + dev.vibe-afrika pcm-kafka-events @@ -55,7 +71,7 @@ net.devh grpc-server-spring-boot-starter - 2.15.0.RELEASE + ${grpc-spring-boot-starter.version} dev.vibe-afrika @@ -73,7 +89,7 @@ net.logstash.logback logstash-logback-encoder - 7.4 + ${logstash-logback-encoder.version} diff --git a/preference-service/src/main/resources/application.yml b/preference-service/src/main/resources/application.yml index 85d85da..4e8de5d 100644 --- a/preference-service/src/main/resources/application.yml +++ b/preference-service/src/main/resources/application.yml @@ -2,4 +2,19 @@ spring: application: name: preference-service config: - import: "optional:configserver:http://localhost:8888" + import: "optional:configserver:${CONFIG_SERVER_URL:http://localhost:8888}" + cloud: + config: + fail-fast: false + # Local fallback for Kafka if config-server is down + stream: + kafka: + binder: + brokers: ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092} + + # Local fallback for Datasource + datasource: + url: jdbc:postgresql://${DB_HOST:localhost}:${DB_PORT:8843}/preference_db + username: ${DB_USERNAME:pcm} + password: ${DB_PASSWORD:pcm_dev_password} + driver-class-name: org.postgresql.Driver diff --git a/profile-service/pom.xml b/profile-service/pom.xml index c8ea113..cf9e040 100644 --- a/profile-service/pom.xml +++ b/profile-service/pom.xml @@ -62,7 +62,7 @@ net.devh grpc-server-spring-boot-starter - 2.15.0.RELEASE + ${grpc-spring-boot-starter.version} @@ -70,6 +70,10 @@ org.springframework.cloud spring-cloud-starter-stream-kafka + + org.springframework.cloud + spring-cloud-stream-binder-rabbit + @@ -89,7 +93,7 @@ net.logstash.logback logstash-logback-encoder - 7.4 + ${logstash-logback-encoder.version} @@ -123,6 +127,10 @@ io.micrometer micrometer-tracing-bridge-otel + + io.opentelemetry + opentelemetry-exporter-otlp + diff --git a/profile-service/src/main/java/dev/vibeafrika/pcm/profile/application/service/PIIProtectionService.java b/profile-service/src/main/java/dev/vibeafrika/pcm/profile/application/service/PIIProtectionService.java index 2b5a77c..2958432 100644 --- a/profile-service/src/main/java/dev/vibeafrika/pcm/profile/application/service/PIIProtectionService.java +++ b/profile-service/src/main/java/dev/vibeafrika/pcm/profile/application/service/PIIProtectionService.java @@ -1,6 +1,6 @@ package dev.vibeafrika.pcm.profile.application.service; -import dev.vibeafrika.pcm.profile.infrastructure.security.VaultTransitService; +import dev.vibeafrika.pcm.common.security.PiiProtectionProvider; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.springframework.stereotype.Service; @@ -17,7 +17,7 @@ @Slf4j public class PIIProtectionService { - private final VaultTransitService vaultTransitService; + private final PiiProtectionProvider piiProtectionProvider; private static final Set PII_KEYS = Set.of("email", "fullName", "phoneNumber"); @@ -33,7 +33,7 @@ public Map protect(Map attributes) { if (protectedAttributes.containsKey(key)) { Object value = protectedAttributes.get(key); if (value instanceof String plaintext) { - String encrypted = vaultTransitService.encrypt(plaintext); + String encrypted = piiProtectionProvider.encrypt(plaintext); log.info("Key {}: plaintext encrypted successfully", key); protectedAttributes.put(key, encrypted); } @@ -54,7 +54,7 @@ public Map unprotect(Map attributes) { if (unprotectedAttributes.containsKey(key)) { Object value = unprotectedAttributes.get(key); if (value instanceof String ciphertext) { - String decrypted = vaultTransitService.decrypt(ciphertext); + String decrypted = piiProtectionProvider.decrypt(ciphertext); log.info("Key {}: ciphertext decrypted successfully", key); unprotectedAttributes.put(key, decrypted); } diff --git a/profile-service/src/main/java/dev/vibeafrika/pcm/profile/domain/model/Profile.java b/profile-service/src/main/java/dev/vibeafrika/pcm/profile/domain/model/Profile.java index ae98e73..d1e4df6 100644 --- a/profile-service/src/main/java/dev/vibeafrika/pcm/profile/domain/model/Profile.java +++ b/profile-service/src/main/java/dev/vibeafrika/pcm/profile/domain/model/Profile.java @@ -5,11 +5,11 @@ import lombok.AccessLevel; import lombok.Getter; import lombok.NoArgsConstructor; -import org.hibernate.annotations.Type; +import org.hibernate.annotations.JdbcTypeCode; +import org.hibernate.type.SqlTypes; import org.springframework.data.annotation.CreatedDate; import org.springframework.data.annotation.LastModifiedDate; import org.springframework.data.jpa.domain.support.AuditingEntityListener; -import io.hypersistence.utils.hibernate.type.json.JsonBinaryType; import org.hibernate.annotations.SQLDelete; import org.hibernate.annotations.SQLRestriction; @@ -45,8 +45,8 @@ public class Profile { @Embedded private Handle handle; - @Type(JsonBinaryType.class) - @Column(name = "attributes", columnDefinition = "jsonb") + @JdbcTypeCode(SqlTypes.JSON) + @Column(name = "attributes") private Map attributes = new HashMap<>(); @CreatedDate diff --git a/profile-service/src/main/java/dev/vibeafrika/pcm/profile/infrastructure/security/PIIProtectionConfig.java b/profile-service/src/main/java/dev/vibeafrika/pcm/profile/infrastructure/security/PIIProtectionConfig.java new file mode 100644 index 0000000..df99a16 --- /dev/null +++ b/profile-service/src/main/java/dev/vibeafrika/pcm/profile/infrastructure/security/PIIProtectionConfig.java @@ -0,0 +1,31 @@ +package dev.vibeafrika.pcm.profile.infrastructure.security; + +import dev.vibeafrika.pcm.common.security.LocalAesPiiProtectionProvider; +import dev.vibeafrika.pcm.common.security.PiiProtectionProvider; +import lombok.extern.slf4j.Slf4j; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.vault.core.VaultOperations; + +@Configuration +@Slf4j +public class PIIProtectionConfig { + + @Bean + @ConditionalOnProperty(name = "pcm.security.pii.provider", havingValue = "vault", matchIfMissing = true) + public PiiProtectionProvider vaultPiiProtectionProvider(VaultOperations vaultOperations, + @Value("${pcm.profile.vault.transit.key-name:pcm-pii-key}") String piiKeyName) { + log.info("Configuring Vault as the PII Protection Provider"); + return new VaultTransitService(vaultOperations, piiKeyName); + } + + @Bean + @ConditionalOnProperty(name = "pcm.security.pii.provider", havingValue = "local") + public PiiProtectionProvider localPiiProtectionProvider( + @Value("${pcm.security.pii.local.secret:abcdefghijklmnop}") String secret) { + log.info("Configuring Local AES as the PII Protection Provider"); + return new LocalAesPiiProtectionProvider(secret); + } +} diff --git a/profile-service/src/main/java/dev/vibeafrika/pcm/profile/infrastructure/security/VaultTransitService.java b/profile-service/src/main/java/dev/vibeafrika/pcm/profile/infrastructure/security/VaultTransitService.java index 432d3a5..d2a9317 100644 --- a/profile-service/src/main/java/dev/vibeafrika/pcm/profile/infrastructure/security/VaultTransitService.java +++ b/profile-service/src/main/java/dev/vibeafrika/pcm/profile/infrastructure/security/VaultTransitService.java @@ -1,5 +1,6 @@ package dev.vibeafrika.pcm.profile.infrastructure.security; +import dev.vibeafrika.pcm.common.security.PiiProtectionProvider; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Service; import org.springframework.vault.core.VaultOperations; @@ -11,7 +12,7 @@ * Provides transparent encryption and decryption for sensitive user data. */ @Service -public class VaultTransitService { +public class VaultTransitService implements PiiProtectionProvider { private final VaultOperations vaultOperations; private final String piiKeyName; diff --git a/profile-service/src/main/resources/application.yml b/profile-service/src/main/resources/application.yml index 8017ac1..e35342c 100644 --- a/profile-service/src/main/resources/application.yml +++ b/profile-service/src/main/resources/application.yml @@ -2,10 +2,22 @@ spring: application: name: profile-service config: - import: "optional:configserver:http://localhost:8888" + import: "optional:configserver:${CONFIG_SERVER_URL:http://localhost:8888}" cloud: config: - fail-fast: false # In dev, we might start services before config-service is healthy + fail-fast: false retry: initial-interval: 2000 max-attempts: 10 + # Local fallback for Kafka if config-server is down + stream: + kafka: + binder: + brokers: ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092} + + # Local fallback for Datasource + datasource: + url: jdbc:postgresql://${DB_HOST:localhost}:${DB_PORT:8843}/profile_db + username: ${DB_USERNAME:pcm} + password: ${DB_PASSWORD:pcm_dev_password} + driver-class-name: org.postgresql.Driver diff --git a/profile-service/src/main/resources/db/migration/V1__create_profiles_table.sql b/profile-service/src/main/resources/db/migration/V1__create_profiles_table.sql index ed3a2f0..bed8584 100644 --- a/profile-service/src/main/resources/db/migration/V1__create_profiles_table.sql +++ b/profile-service/src/main/resources/db/migration/V1__create_profiles_table.sql @@ -3,21 +3,18 @@ CREATE TABLE profiles ( id UUID PRIMARY KEY, tenant_id VARCHAR(100) NOT NULL, handle VARCHAR(50) NOT NULL UNIQUE, - attributes JSONB, - created_at TIMESTAMP WITH TIME ZONE NOT NULL, - updated_at TIMESTAMP WITH TIME ZONE NOT NULL, + attributes JSON, + created_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NOT NULL, version BIGINT NOT NULL DEFAULT 0, - deleted_at TIMESTAMP WITH TIME ZONE + deleted_at TIMESTAMP ); -- Create indexes CREATE INDEX idx_profile_tenant ON profiles(tenant_id); CREATE INDEX idx_profile_handle ON profiles(handle); CREATE INDEX idx_profile_created_at ON profiles(created_at DESC); -CREATE INDEX idx_profile_deleted_at ON profiles(deleted_at) WHERE deleted_at IS NULL; - --- Create GIN index for JSONB attributes for faster queries -CREATE INDEX idx_profile_attributes ON profiles USING GIN (attributes); +CREATE INDEX idx_profile_deleted_at ON profiles(deleted_at); -- Add comments COMMENT ON TABLE profiles IS 'User profiles with dynamic attributes stored as JSONB'; diff --git a/segment-service/pom.xml b/segment-service/pom.xml index 718d4ab..b149ac4 100644 --- a/segment-service/pom.xml +++ b/segment-service/pom.xml @@ -29,6 +29,18 @@ org.springframework.boot spring-boot-starter-actuator + + io.micrometer + micrometer-registry-prometheus + + + io.micrometer + micrometer-tracing-bridge-otel + + + io.opentelemetry + opentelemetry-exporter-otlp + org.springframework.cloud spring-cloud-starter-config @@ -49,6 +61,10 @@ org.springframework.cloud spring-cloud-starter-stream-kafka + + org.springframework.cloud + spring-cloud-stream-binder-rabbit + dev.vibe-afrika pcm-kafka-events @@ -59,7 +75,7 @@ net.devh grpc-server-spring-boot-starter - 2.15.0.RELEASE + ${grpc-spring-boot-starter.version} dev.vibe-afrika @@ -77,7 +93,7 @@ net.logstash.logback logstash-logback-encoder - 7.4 + ${logstash-logback-encoder.version} diff --git a/segment-service/src/main/resources/application.yml b/segment-service/src/main/resources/application.yml index 0c99b59..16db8b9 100644 --- a/segment-service/src/main/resources/application.yml +++ b/segment-service/src/main/resources/application.yml @@ -2,4 +2,19 @@ spring: application: name: segment-service config: - import: "optional:configserver:http://localhost:8888" + import: "optional:configserver:${CONFIG_SERVER_URL:http://localhost:8888}" + cloud: + config: + fail-fast: false + # Local fallback for Kafka if config-server is down + stream: + kafka: + binder: + brokers: ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092} + + # Local fallback for Datasource + datasource: + url: jdbc:postgresql://${DB_HOST:localhost}:${DB_PORT:8843}/segment_db + username: ${DB_USERNAME:pcm} + password: ${DB_PASSWORD:pcm_dev_password} + driver-class-name: org.postgresql.Driver