Docs: Improve modules/jenkins/README.md: Add context, architecture decisions, and cost guidance

[gabebatista]

What were you searching in the docs?

Context on why to deploy Jenkins in the cloud, what the architecture decisions mean, prerequisites explanations, cost estimates, and production considerations.

Is this related to an existing documentation section?

modules/jenkins/README.md - proposing new sections to add context around the existing technical documentation.

How can we improve?

The Jenkins README has good technical documentation but lacks context on WHY users should deploy Jenkins in the cloud, WHY specific architecture decisions were made, WHAT the costs will be, and WHAT to consider for production. Adding these sections helps users make informed decisions and justify the architecture to stakeholders.

Got a suggestion in mind?

1. Add "Why Deploy Jenkins on AWS?" Section (After opening paragraph)

## Why Deploy Jenkins on AWS?

**vs On-Premises Jenkins**:
- **No hardware management**: No servers to rack, power, cool, or maintain
- **Elastic scaling**: Build agents scale up during peak hours, scale down overnight
- **Pay-per-use**: Only pay for compute during builds, not idle capacity
- **AWS service integration**: Native access to S3, Secrets Manager, ECR, CodeArtifact
- **High availability**: Multi-AZ deployment without complex hardware redundancy
- **Disaster recovery**: Infrastructure defined as code, redeploy in minutes

**vs Jenkins on EC2 (Traditional AWS Deployment)**:
- **No server management**: ECS Fargate eliminates EC2 patching and maintenance
- **Dynamic scaling**: Agents spin up on-demand for builds
- **Cost efficiency**: No paying for idle EC2 instances between builds

**Best For**:
- Teams moving from on-premises to cloud infrastructure
- Studios with variable build workloads (nights/weekends idle)
- Organizations requiring compliance (VPC isolation, encrypted storage)
- Teams wanting infrastructure-as-code for Jenkins deployment

---

2. Enhance Prerequisites with Why Explanations

Replace the current prerequisites list with a detailed table:

## Prerequisites

### Required AWS Infrastructure

| Resource | Why It's Needed | How to Verify |
|----------|-----------------|---------------|
| **VPC with subnets** | Module deploys into existing network infrastructure | \`aws ec2 describe-vpcs\` |
| **Public subnets** | For ALB to receive internet traffic | \`aws ec2 describe-subnets --filters "Name=map-public-ip-on-launch,Values=true"\` |
| **Private subnets** | For ECS tasks and EFS (security best practice) | \`aws ec2 describe-subnets --filters "Name=map-public-ip-on-launch,Values=false"\` |
| **Route53 hosted zone** | For DNS record creation (e.g., jenkins.example.com) | \`aws route53 list-hosted-zones\` |
| **ACM certificate** | For HTTPS termination at ALB | \`aws acm list-certificates\` |
| **NAT Gateway** | For ECS tasks to pull Docker images and access AWS APIs | \`aws ec2 describe-nat-gateways\` |

### Custom AMI (Optional but Recommended)

**Why custom AMI**: Pre-baked Jenkins configuration, plugins, and tools reduce container startup time from 5 minutes to 30 seconds.

**Build with Packer**:
\`\`\`bash
cd assets/packer/jenkins
packer build jenkins.pkr.hcl  # ~15 minutes
\`\`\`

**Alternative**: Use default AWS ECS-optimized AMI (slower startup, requires runtime plugin installation)

### Tools and Permissions

- **Terraform >= 1.10.3**: Module uses modern Terraform features
- **AWS CLI**: For post-deployment tasks (secret retrieval, log viewing)
- **IAM permissions**: EC2, ECS, EFS, ALB, Route53, Secrets Manager, CloudWatch

**Time Estimate**: 30-60 minutes to complete all prerequisites.

---

3. Add "Architecture Decisions" Section (After architecture diagram)

## Architecture Decisions

### Why ECS Fargate?
- **No server management**: AWS handles underlying compute infrastructure
- **Cost efficiency**: Pay per second of build execution, not idle time
- **Auto-scaling**: Scale build agents based on queue depth

**Alternative Considered**: Jenkins on EC2 with ASG
**Why Not Used**: Requires AMI management, patching, and doesn't scale to zero

### Why EFS for Jenkins Home?
- **Persistence**: Jenkins configuration and jobs survive container restarts
- **Multi-AZ durability**: Automatic replication across availability zones
- **Shared access**: Multiple ECS tasks can mount same filesystem

**Alternative Considered**: EBS volumes
**Why Not Used**: EBS requires EC2 instances (not compatible with Fargate)

### Why ALB Instead of NLB?
- **HTTPS termination**: ALB handles SSL/TLS, Jenkins container runs HTTP
- **Path-based routing**: Can route to multiple services from one ALB (future extensibility)
- **Health checks**: Application-level health checks (HTTP 200)

**Alternative Considered**: NLB
**Why Not Used**: Requires TLS configuration within Jenkins container

### Why Private Subnets for ECS Tasks?
- **Security**: Jenkins has no direct internet exposure
- **Compliance**: Meets requirements for production CI/CD systems
- **Controlled egress**: All outbound traffic via NAT Gateway (auditable)

**Alternative Considered**: Public subnets
**Why Not Used**: Increases attack surface, violates least-privilege principle

### Why Secrets Manager for Credentials?
- **Rotation**: Automatic credential rotation without Jenkins restart
- **Audit**: CloudTrail logs all secret access
- **IAM integration**: Fine-grained permissions for secret access

**Alternative Considered**: Jenkins credentials store
**Why Not Used**: Harder to rotate, no automatic rotation, requires Jenkins restart

---

4. Add Cost Estimation Section (Before "Getting Started")

## Cost Considerations

⚠️ **Jenkins infrastructure costs vary based on usage. Typical range: $200-500/month.**

### Cost Breakdown (us-east-1)

| Component | Configuration | Typical Cost/Month | Notes |
|-----------|---------------|------------|-------|
| **ECS Fargate Controller** | 1 vCPU, 2GB RAM, 24/7 | ~$30 | Jenkins controller |
| **ECS Fargate Agents** | 4 vCPU, 8GB RAM, variable | ~$100-300 | Build agents (usage-based) |
| **Application Load Balancer** | 1 ALB, 24/7 | ~$20 | HTTPS termination |
| **EFS** | 100GB storage | ~$30 | Jenkins home directory |
| **RDS (if used)** | db.t3.medium, 100GB storage | ~$100 | PostgreSQL for plugins |
| **Data Transfer** | 100GB outbound | ~$9 | Artifact downloads |
| **CloudWatch Logs** | 10GB ingested, 30-day retention | ~$5 | Log storage |

**Typical Monthly Range**: 
- **Basic usage** (few builds): ~$200/month
- **Moderate usage** (regular builds): ~$350/month  
- **Heavy usage** (continuous builds): ~$500+/month

### Cost Optimization

1. **Stop Jenkins outside business hours**:
   \`\`\`bash
   # Stop Jenkins (scale to 0)
   aws ecs update-service --cluster jenkins --service jenkins-controller --desired-count 0
   \`\`\`
   **Potential Savings**: ~$20/month if stopped 16 hours/day on weekdays

2. **Use Spot Instances for build agents**:
   - ECS Fargate Spot provides up to 70% discount for interruptible workloads
   - Suitable for builds that can tolerate interruptions
   **Potential Savings**: Variable based on agent usage

3. **Reduce EFS throughput mode**:
   - Use "Bursting" mode instead of "Provisioned" for low-activity Jenkins
   **Potential Savings**: ~$20/month

4. **Optimize CloudWatch log retention**:
   - Reduce retention from 30 days to 7 days for non-production
   **Potential Savings**: ~$3/month

**Use [AWS Pricing Calculator](https://calculator.aws) for accurate estimates based on your specific usage patterns**.

---

5. Add "Production Considerations" Section (Before "Getting Started")

## Production Considerations

When preparing to deploy this module in a production environment, consider the following:

### Security
- Review and restrict \`allowed_cidr_blocks\` to specific IP ranges (avoid 0.0.0.0/0)
- Enable MFA for AWS IAM users with access to Jenkins infrastructure
- Configure VPC Flow Logs for network traffic auditing
- Implement secret rotation policies for Secrets Manager credentials
- Enable CloudTrail for API activity logging

### High Availability & Reliability
- Deploy across multiple Availability Zones
- Configure auto-scaling policies for build agents
- Enable EFS backups for Jenkins home directory
- Test disaster recovery procedures (restoring from EFS backup)
- Document runbooks for common failure scenarios

### Monitoring & Observability
- Set up CloudWatch alarms for critical metrics (CPU, memory, disk usage)
- Configure billing alerts for unexpected cost increases
- Implement centralized log aggregation
- Define and monitor SLAs for build completion times
- Set up alerting for failed builds and infrastructure issues

### Performance
- Right-size ECS task configurations based on actual workload metrics
- Monitor EFS throughput and adjust mode if needed (Bursting vs Provisioned)
- Consider using EFS Provisioned Throughput for high-concurrency builds
- Review and optimize Jenkins plugin configuration

### Operations
- Document procedures for common operations (adding users, scaling agents, upgrades)
- Establish backup and retention policies for build artifacts
- Plan for Jenkins version upgrades and plugin updates
- Define incident response procedures
- Train operations team on AWS Console access and troubleshooting

[gabebatista]

📋 README Structure Template

To maintain consistency across all module READMEs, we're working towards a standardized structure. While we want to stick as closely to this template as possible, it's okay to make changes to fit the specific module's needs.

Template Structure:

# [Module/Service Name]

> **One-sentence description**: Brief, clear statement of what this module does and its primary purpose.

## Table of Contents

- [Overview](#overview)
- [Why This Module?](#why-this-module)
- [Architecture](#architecture)
- [Cost Considerations](#cost-considerations)
- [Production Considerations](#production-considerations)
- [Quick Start](#quick-start)
- [Usage Examples](#usage-examples)
- [Configuration Reference](#configuration-reference)
- [Operations](#operations)
- [Troubleshooting](#troubleshooting)

## Overview

Brief 2-3 paragraph introduction covering:
- What this module deploys (high-level components)
- Primary use case(s)
- Target audience (game studios, indie developers, enterprise teams, etc.)

## Why This Module?

**Note**: For modules with submodules (e.g., Perforce with P4 Server, P4 Auth, P4 Code Review), add a "Module Components" section here explaining what each component does and when to deploy them.

### Problem Statement
What problem does this solve for game development teams?

### Why [This Tool/Service]?
- **vs Alternative 1**: Key differentiator (e.g., "Perforce vs Git: Binary file handling")
- **vs Alternative 2**: Key differentiator (e.g., "Self-hosted vs SaaS: Control and compliance")
- **vs Alternative 3**: Key differentiator (e.g., "TeamCity vs Jenkins: UI polish vs flexibility")
- **Best for**: Team size, project type, specific requirements

**Note**: For modules with multiple alternatives (e.g., CI/CD options), include comparison tables in this section:

| Factor | This Module | Alternative Module |
|--------|-------------|-------------------|
| **Factor 1** | Advantage | Trade-off |
| **Factor 2** | Trade-off | Advantage |

### When NOT to Use This Module
- Small teams (< X people) - simpler alternatives may suffice
- Text-only repositories - Git may be more cost-effective
- Teams without [specific requirement] - overhead not justified

## Architecture

### High-Level Architecture

\`\`\`
[Include diagram - keep it simple and focused on data flow]
\`\`\`

**Key Components:**
- **Component 1**: One-line purpose
- **Component 2**: One-line purpose
- **Component 3**: One-line purpose

### Architecture Decisions

> **Why these choices matter**: Understanding the rationale helps you customize safely and explain to stakeholders.

Use collapsible <details> sections to keep README digestible:

<details>
<summary><b>Decision 1: Example Decision</b></summary>

**Choice**: What was chosen

**Why**:
- Reason 1
- Reason 2
- Reason 3

**Trade-off**:
- What you give up with this choice

**When to override**: When this choice might not fit your needs
</details>

## Cost Considerations

### Estimated Monthly Cost

**Small Team (5-10 developers)**
| Component | Configuration | Cost |
|-----------|--------------|------|
| Compute | 2x t3.medium (24/7) | $60 |
| Storage | 500GB EBS | $50 |
| Network | NLB + 1TB transfer | $40 |
| **Total** | | **~$150/month** |

**Cost Optimization:**
- Schedule dev environments to stop nights/weekends
- Use Savings Plans for steady-state workloads
- Right-size after 2 weeks based on metrics

**Use [AWS Pricing Calculator](https://calculator.aws) for accurate estimates**.

## Production Considerations

> **Note**: This section provides guidance on things to consider when preparing for production deployment. It's ultimately up to you to determine what's appropriate for your specific environment and requirements.

When preparing to deploy this module in a production environment, consider the following:

### Security
- Review and restrict access controls (security groups, CIDR blocks)
- Enable MFA for AWS IAM users with infrastructure access
- Configure VPC Flow Logs for network traffic auditing
- Implement secret rotation policies where applicable
- Enable CloudTrail for API activity logging
- Review IAM permissions for least privilege access

### High Availability & Reliability
- Deploy across multiple Availability Zones where supported
- Enable automated backups with appropriate retention periods
- Test disaster recovery procedures (restoring from backups)
- Document runbooks for common failure scenarios

### Monitoring & Observability
- Set up CloudWatch alarms for critical metrics
- Configure billing alerts for unexpected cost increases
- Implement centralized log aggregation
- Define and monitor relevant SLAs
- Set up alerting for service degradation or failures

### Performance
- Right-size resources based on actual workload metrics
- Monitor performance metrics and adjust configurations as needed
- Plan for capacity growth as team/usage scales

### Operations
- Document procedures for common operations
- Establish backup and retention policies
- Plan for version upgrades and patching
- Define incident response procedures
- Train operations team on troubleshooting and maintenance

Target Length

500-800 lines (excluding terraform-docs) to keep documentation digestible.