Implement hot spare configuration for Keycloak high availability

[Copilot]

Adds hot spare functionality to provide high availability for the critical Keycloak authentication service.

Overview

This implementation adds a secondary Keycloak instance that runs alongside the primary instance, sharing the same database. Apache load balancing with health checks provides automatic failover if the primary instance becomes unavailable.

Key Features

Hot Spare Configuration

• Added ENABLE_HOT_SPARE and KC_SPARE_HOST_PORT configuration variables
• Second Keycloak instance runs on port 3124 (configurable)
• Both instances share the same PostgreSQL database for seamless failover
• Docker Compose profiles control when hot spare runs

Automatic Failover

• Apache mod_proxy_balancer provides load balancing between instances
• Health checks monitor both primary (port 3123) and spare (port 3124) instances
• Balancer manager interface available at /health/balancer-manager for monitoring
• Automatic retry and ping configuration for robust failover

Management Tools

• ./cmds/manage-hot-spare.sh - Enable/disable hot spare configuration
• ./cmds/health-check.sh - Monitor health of all instances
• Updated deployment documentation with hot spare procedures

Usage

# Enable hot spare
./cmds/manage-hot-spare.sh enable

# Deploy with hot spare
docker compose --profile hot-spare up -d

# Deploy without hot spare (maintains backward compatibility)
docker compose up -d

# Monitor health
./cmds/health-check.sh

Benefits

• High Availability: Automatic failover ensures service continuity
• Zero Downtime: Seamless switching between instances
• Backward Compatible: Existing single-instance deployments continue to work
• Easy Management: Simple commands to enable/disable hot spare
• Monitoring: Built-in health checks for proactive monitoring

Fixes #8.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[jrmerz]

@copilot can you set it up so there are two docker compose clusters running on two separate vms. The backup postgres instance will be a read only replicate of the primary. Additionally, can the replicate keycloak UI be set to be read only?

[Copilot]

@copilot can you set it up so there are two docker compose clusters running on two separate vms. The backup postgres instance will be a read only replicate of the primary. Additionally, can the repl...

I'll implement the distributed architecture you've requested. This involves creating separate Docker Compose configurations for primary and replica VMs, setting up PostgreSQL streaming replication, and configuring the replica Keycloak instance to be read-only. This will provide true high availability across separate infrastructure. Let me work on these changes.