Feature request: Consolidate Unreal Cloud DDC Modules

[novekm]

Use case

Problem Statement

The current Unreal Cloud DDC uses a two-module architecture that creates unnecessary complexity:

1. unreal-cloud-ddc-infra - AWS resources (EKS, ScyllaDB, networking)
2. unreal-cloud-ddc-intra-cluster - Kubernetes/Helm deployments

Current Pain Points:

• 400+ lines of boilerplate in multi-region samples
• Complex provider management (AWS, AWSCC, Kubernetes, Helm per region)
• Manual dependency coordination between modules
• High barrier to entry - users must understand EKS, ScyllaDB, cross-region networking
• Inconsistent with Perforce module which provides single, simple interface

Current Multi-Region Usage:

# Complex providers.tf (80+ lines)
provider "aws" { alias = "region-1"; region = var.regions[0] }
provider "aws" { alias = "region-2"; region = var.regions[1] }
provider "awscc" { alias = "region-1"; region = var.regions[0] }
provider "awscc" { alias = "region-2"; region = var.regions[1] }
# ... complex Kubernetes/Helm providers with EKS dependencies

# Complex main.tf (400+ lines)
module "unreal_cloud_ddc_infra_region_1" { /* 30+ lines */ }
module "unreal_cloud_ddc_intra_cluster_region_1" { /* 20+ lines */ }
module "unreal_cloud_ddc_infra_region_2" { /* 30+ lines */ }
module "unreal_cloud_ddc_intra_cluster_region_2" { /* 20+ lines */ }
# ... VPC setup, security groups, cross-region networking

Solution/User Experience

Proposed Solution

Create unified parent module that encapsulates complexity while preserving existing logic as submodules.

Expected User Experience:

# Simple providers.tf
provider "aws" { region = "us-east-1" }
provider "aws" { alias = "us-east-2"; region = "us-east-2" }

# Simple main.tf
module "unreal_cloud_ddc" {
  source = "../../modules/unreal/unreal-cloud-ddc"
  
  regions = {
    primary = { region = "us-east-1" }
    secondary = { region = "us-east-2" }
  }
  vpc_ids = { primary = aws_vpc.primary.id, secondary = aws_vpc.secondary.id }
  providers = { aws.primary = aws, aws.secondary = aws.us-east-2 }
}

Module Structure

modules/unreal/unreal-cloud-ddc/
├── main.tf                          # Parent module (user-facing)
├── variables.tf
├── outputs.tf
├── versions.tf
└── modules/
    ├── infrastructure/              # AWS resources (EKS, ScyllaDB)
    └── applications/                # Kubernetes/Helm deployments

How It Works

Conditional Module Instantiation: Parent module uses count to conditionally create resources:

# Inside parent module
locals {
  is_multi_region = contains(keys(var.regions), "secondary")
  primary_region  = var.regions.primary.region
  secondary_region = try(var.regions.secondary.region, null)
}

# Always create primary region
module "infrastructure_primary" {
  source = "./modules/infrastructure"
  providers = { aws = aws.primary, awscc = awscc.primary }
}

module "applications_primary" {
  source = "./modules/applications"
  providers = { kubernetes = kubernetes.primary, helm = helm.primary }
  depends_on = [module.infrastructure_primary]
}

# Conditionally create secondary region
module "infrastructure_secondary" {
  count = local.is_multi_region ? 1 : 0
  source = "./modules/infrastructure"
  providers = { aws = aws.secondary, awscc = awscc.secondary }
}

module "applications_secondary" {
  count = local.is_multi_region ? 1 : 0
  source = "./modules/applications"
  providers = { kubernetes = kubernetes.secondary, helm = helm.secondary }
  depends_on = [module.infrastructure_secondary]
}

Provider Abstraction: Module handles all complex provider setup (AWSCC, Kubernetes, Helm) internally. Users only provide simple AWS providers.

Benefits

• 90% reduction in boilerplate (400+ lines → ~50 lines)
• Provider complexity abstraction - no AWSCC, Kubernetes, Helm management
• Clear dependency chain - applications depends on infrastructure
• Consistent with Perforce module pattern
• Backward compatibility - existing modules remain available
• Flexible networking - works with existing VPCs
• Standard Terraform workflow - single apply/destroy

Implementation Plan

Phase 1: Create Parent Module Structure

1. Create new directory structure:

   modules/unreal/unreal-cloud-ddc/
   ├── main.tf
   ├── variables.tf
   ├── outputs.tf
   ├── versions.tf
   └── modules/
       ├── infrastructure/     # Move existing unreal-cloud-ddc-infra here
       └── applications/       # Move existing unreal-cloud-ddc-intra-cluster here
   

2. Move existing modules:

   • Rename unreal-cloud-ddc-infra/ → modules/infrastructure/
   • Rename unreal-cloud-ddc-intra-cluster/ → modules/applications/
   • Update any internal references

Phase 2: Implement Parent Module Logic

3. Create versions.tf with all required providers:

   terraform {
     required_providers {
       aws = {
         source = "hashicorp/aws"
         version = ">= 6.2.0"
         configuration_aliases = [aws.primary, aws.secondary]
       }
       awscc = { source = "hashicorp/awscc", version = ">= 1.26.0" }
       kubernetes = { source = "hashicorp/kubernetes", version = ">= 2.24.0" }
       helm = { source = "hashicorp/helm", version = "2.17.0" }
     }
   }

4. Create variables.tf with simplified interface:

   variable "regions" {
     description = "Map of regions to deploy to"
     type = map(object({
       region = string
       # Future: availability_zones, vpc_cidr, etc.
     }))
     validation {
       condition = contains(keys(var.regions), "primary")
       error_message = "Must specify a primary region."
     }
   }

   • github_credential_arns (list matching regions)
   • Optional customization variables (instance types, storage, etc.)

5. Implement main.tf with:

   • Local variables for multi-region detection
   • Internal provider configurations (AWSCC, Kubernetes, Helm)
   • Conditional module instantiation using count
   • Cross-region networking setup (VPC peering between existing VPCs)
   • Token management and replication

6. Create outputs.tf exposing key resources:

   • EKS cluster endpoints
   • ScyllaDB connection details
   • Load balancer URLs
   • VPC information

Phase 3: Update Samples and Documentation

7. Update multi-region sample to use new unified module:

   • Keep VPC creation at sample level (following Perforce pattern)
   • Replace complex module orchestration with single module call
   • Reduce from 400+ lines to ~50 lines (VPC + module call)
   • Simple provider configuration

8. Create single-region sample showing minimal usage

9. Update documentation:

   • Migration guide from existing modules
   • New simplified usage examples
   • Architecture diagrams showing parent/submodule relationship

Phase 4: Testing and Validation

10. Test deployment scenarios:

    • Single region deployment
    • Multi-region deployment
    • Terraform destroy (ensure cleanup works)
    • Provider edge cases

11. Validate backward compatibility:

    • Ensure existing individual modules still work
    • Test existing samples continue to function

Phase 5: Migration Support

12. Create migration documentation:

    • Step-by-step guide for existing users
    • Comparison of before/after configurations
    • Troubleshooting common migration issues

13. Consider deprecation timeline for individual modules:

    • Mark as "legacy" but maintain for backward compatibility
    • Encourage new users to use unified module
    • Plan eventual removal in future major version

Success Criteria

• ✅ Single module call deploys complete DDC infrastructure
• ✅ 90%+ reduction in user configuration complexity
• ✅ Backward compatibility maintained
• ✅ Standard terraform apply/destroy workflow
• ✅ Clear documentation and migration path
• ✅ Consistent with other CGD Toolkit modules (Perforce pattern)

Expected Timeline: 2-3 weeks for implementation, 1 week for testing and documentation.

Alternative solutions

I'll help update the feature request based on the context provided. Here's a suggested update:

Additional Notes:

1. BLOCKING: PR feat: Unreal Cloud DDC Multi-Region  #671 should not be merged until this feature request is addressed, as it will impact the changes proposed in that PR.

2. Directory Structure Correction:

   • Current: Content is incorrectly placed in 'samples' directory
   • Required: Should be moved to 'examples' directory
   • Rationale: Unreal Cloud DDC is the only module that lacks an 'examples' directory and incorrectly uses 'samples'

3. Architectural Clarification:

   • Examples: Should demonstrate single module usage with various configurations
   • Samples: Should demonstrate integration of multiple modules together

4. The planned module consolidation and restructuring will help:

   • Maintain consistency across all modules
   • Properly organize examples vs samples
   • Establish clear separation of concerns

No response

[novekm]

some changes made locally to ddc (not pushed yet) that likely should be added in the consolidation process:

Unreal Cloud DDC Multi-Region Changes

Overview

This document outlines the changes made to the Unreal Cloud DDC multi-region implementation to improve reliability, flexibility, and cleanup handling.

1. EKS Cluster Timeout Configuration

File: modules/unreal/unreal-cloud-ddc/unreal-cloud-ddc-infra/eks.tf

Why: EKS cluster operations can take longer than Terraform's default timeouts, especially in multi-region deployments or during high-load periods. This prevents timeout failures during cluster lifecycle operations.

Change:

resource "aws_eks_cluster" "unreal_cloud_ddc_eks_cluster" {
  # ... existing configuration ...
  
  # NEW: Added timeout configuration
  timeouts {
    create = "30m"
    update = "60m"
    delete = "30m"
  }
}

2. Helm Release Reliability Improvements

File: modules/unreal/unreal-cloud-ddc/unreal-cloud-ddc-intra-cluster/helm.tf

Why: Helm deployments can fail due to webhook issues or leave orphaned resources. These changes improve deployment reliability and ensure proper cleanup.

Changes:

Helm Release Configuration

resource "helm_release" "unreal_cloud_ddc_initialization" {
  # ... existing configuration ...
  
  # NEW: Improve deployment reliability
  disable_webhooks = true
  cleanup_on_fail  = true
  
  depends_on = [
    kubernetes_service_account.unreal_cloud_ddc_service_account,
    kubernetes_namespace.unreal_cloud_ddc,
    aws_ecr_pull_through_cache_rule.unreal_cloud_ddc_ecr_pull_through_cache_rule,
    null_resource.cleanup_on_destroy  # NEW: Ensure cleanup resource exists
  ]
}

Kubernetes Cleanup on Destroy

# NEW: Added comprehensive cleanup resource
resource "null_resource" "cleanup_on_destroy" {
  triggers = {
    cluster_name = data.aws_eks_cluster.unreal_cloud_ddc_cluster.name
    region      = var.region
    namespace   = var.unreal_cloud_ddc_namespace
  }

  provisioner "local-exec" {
    when    = destroy
    command = <<-EOT
      # Only run cleanup if we can connect to the cluster
      if aws eks update-kubeconfig --region ${self.triggers.region} --name ${self.triggers.cluster_name} 2>/dev/null; then
        echo "Cleaning up Kubernetes resources in namespace ${self.triggers.namespace}"
        kubectl delete all --all -n ${self.triggers.namespace} --timeout=60s --ignore-not-found=true || true
        kubectl delete pvc --all -n ${self.triggers.namespace} --timeout=30s --ignore-not-found=true || true
        kubectl delete namespace ${self.triggers.namespace} --timeout=60s --ignore-not-found=true || true
      else
        echo "Cluster not accessible, skipping Kubernetes cleanup"
      fi
    EOT
  }
}

Why: Ensures proper cleanup of Kubernetes resources during terraform destroy, preventing orphaned resources that could cause issues in subsequent deployments.

3. Sample Configuration Flexibility

Files: samples/unreal-cloud-ddc-multi-region/

Why: The sample was hardcoded to specific regions and required manual configuration. These changes make it more flexible and provide sensible defaults for easier testing.

Variable Consolidation and Defaults

File: variables.tf

# MOVED: From dns.tf to variables.tf with default
variable "route53_public_hosted_zone_name" {
  type        = string
  description = "The root domain name for the Hosted Zone where the ScyllaDB monitoring record should be created."
  default     = "novekm.people.aws.dev"  # NEW: Added default
}

# NEW: Added default values for easier testing
variable "github_credential_arn_region_1" {
  type        = string
  sensitive   = true
  description = "Github Credential ARN"
  default     = "arn:aws:secretsmanager:us-east-1:111111111111:secret:ecr-pullthroughcache/-123xx"
}

variable "github_credential_arn_region_2" {
  type        = string
  sensitive   = true
  description = "Github Credential ARN"
  default     = "arn:aws:secretsmanager:us-east-2:111111111111:secret:ecr-pullthroughcache/-123xx"
}

# CHANGED: Updated default regions
variable "regions" {
  type        = list(string)
  default     = ["us-east-1", "us-east-2"]  # Was: ["us-west-2", "us-east-2"]
  description = "List of regions to deploy the solution"
}

Dynamic Provider Configuration

File: providers.tf

# CHANGED: Made regions dynamic instead of hardcoded
provider "awscc" {
  alias  = "region-1"
  region = var.regions[0]  # Was: "us-west-2"
}

provider "awscc" {
  alias  = "region-2"
  region = var.regions[1]  # Was: "us-east-2"
}

provider "aws" {
  alias  = "region-1"
  region = var.regions[0]  # Was: "us-west-2"
}

provider "aws" {
  alias  = "region-2"
  region = var.regions[1]  # Was: "us-east-2"
}

Improved Availability Zone Handling

File: local.tf

# CHANGED: Made AZ data sources dynamic and more robust
data "aws_availability_zones" "available_region_1" {
  region = var.regions[0]  # Was: "us-west-2"
  
  # NEW: Added filter for opt-in status
  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
  # Commented out hardcoded exclusions for flexibility
}

data "aws_availability_zones" "available_region_2" {
  region = var.regions[1]  # Was: "us-east-2"
  
  # NEW: Added filter for opt-in status
  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

Why: The opt-in-status filter ensures only standard AZs are used, avoiding issues with Local Zones or Wavelength zones that may not support all required services.

Summary of Benefits

1. Reliability: EKS timeouts and Helm configurations prevent deployment failures
2. Cleanup: Proper Kubernetes resource cleanup prevents orphaned resources
3. Flexibility: Dynamic region configuration allows easy testing in different regions
4. Usability: Default values reduce required configuration for testing
5. Robustness: AZ filtering ensures compatible availability zones are selected

Migration Notes

• The sample now defaults to us-east-1 and us-east-2 instead of us-west-2 and us-east-2
• Default values are provided for testing, but should be overridden for production use
• The cleanup resource ensures proper teardown of Kubernetes resources during destroy operations