Docs: Improve modules/teamcity/README.md: Add context, comparison, architecture decisions, and costs

[gabebatista]

What were you searching in the docs?

Context on why to use TeamCity, when to choose TeamCity vs Jenkins, architecture decisions, cost estimates, and production considerations.

Is this related to an existing documentation section?

modules/teamcity/README.md - proposing new sections to add context around the existing technical documentation.

How can we improve?

The TeamCity README has good technical documentation but lacks context on WHY to use TeamCity, WHEN to choose it over Jenkins, WHY specific architecture decisions were made, WHAT the costs are, and WHAT to consider for production. Adding these sections helps users make informed decisions.

Got a suggestion in mind?

1. Add "Why TeamCity on AWS?" Section (After opening paragraph)

## Why TeamCity on AWS?

**vs On-Premises TeamCity**:
- **No hardware management**: No servers to rack, power, cool, or maintain
- **Elastic scaling**: Build agents scale up during peak hours, scale down overnight
- **Pay-per-use**: Only pay for compute during builds, not idle capacity
- **AWS service integration**: Native access to S3, Secrets Manager, ECR, CodeArtifact
- **High availability**: Multi-AZ deployment without complex hardware redundancy
- **Disaster recovery**: Infrastructure defined as code, redeploy in minutes

**vs TeamCity on EC2 (Traditional AWS Deployment)**:
- **No server management**: ECS Fargate eliminates EC2 patching and maintenance
- **Dynamic scaling**: Agents spin up on-demand for builds
- **Cost efficiency**: No paying for idle EC2 instances between builds

**Best For**:
- Teams moving from on-premises to cloud infrastructure
- Studios with variable build workloads (nights/weekends idle)
- Organizations requiring compliance (VPC isolation, encrypted storage)
- Teams wanting infrastructure-as-code for TeamCity deployment

---

2. Add "TeamCity vs Jenkins" Section (After "Why TeamCity on AWS?")

## TeamCity vs Jenkins

Both modules deploy CI/CD infrastructure on AWS using ECS Fargate. Choose based on your team's needs:

| Factor | TeamCity (This Module) | Jenkins Module |
|--------|----------------------|----------------|
| **Licensing** | Requires TeamCity license | Open source, free |
| **User Interface** | Polished, modern UI out-of-box | Plugin-dependent, varies |
| **Setup Complexity** | Lower (good defaults) | Higher (more configuration needed) |
| **Customization** | Good built-in features | Extremely flexible via plugins |
| **Learning Curve** | Lower (intuitive UI) | Moderate (large plugin ecosystem) |
| **Build Configuration** | Kotlin DSL or UI | Jenkinsfile (Groovy) |
| **Game Engine Support** | Built-in Unity/Unreal support | Requires plugin configuration |

**Choose TeamCity if**:
- ✅ You have existing TeamCity licenses
- ✅ Team prefers polished UI over extensive customization
- ✅ Want built-in Unity/Unreal build support
- ✅ Lower learning curve is priority

**Choose Jenkins if**:
- ✅ Cost-sensitive (no licensing fees)
- ✅ Need maximum flexibility and customization
- ✅ Team has Jenkins expertise
- ✅ Extensive plugin ecosystem is valuable

**Note**: Both modules use the same ECS Fargate architecture, so infrastructure costs are similar (excluding TeamCity licensing). Studios with multiple projects may use one CI/CD system for all projects.

---

3. Add "Architecture Decisions" Section (After architecture diagram)

## Architecture Decisions

### Why ECS Fargate?
- **No server management**: AWS handles underlying compute infrastructure
- **Cost efficiency**: Pay per second of build execution, not idle time
- **Auto-scaling**: Scale build agents based on queue depth

**Alternative Considered**: TeamCity on EC2 with ASG
**Why Not Used**: Requires AMI management, patching, and doesn't scale to zero

### Why RDS for TeamCity Database?
- **Managed service**: AWS handles backups, patching, and high availability
- **Multi-AZ**: Automatic failover for production deployments
- **Performance**: Dedicated database instance for TeamCity metadata

**Alternative Considered**: Database on ECS container
**Why Not Used**: Not recommended by JetBrains for production, no easy backup/restore

### Why EFS for TeamCity Data Directory?
- **Persistence**: TeamCity configuration and build artifacts survive container restarts
- **Multi-AZ durability**: Automatic replication across availability zones
- **Shared access**: Multiple ECS tasks can mount same filesystem

**Alternative Considered**: EBS volumes
**Why Not Used**: EBS requires EC2 instances (not compatible with Fargate)

### Why ALB Instead of NLB?
- **HTTPS termination**: ALB handles SSL/TLS, TeamCity container runs HTTP
- **Path-based routing**: Can route to multiple services from one ALB
- **Health checks**: Application-level health checks (HTTP 200)

**Alternative Considered**: NLB
**Why Not Used**: Requires TLS configuration within TeamCity container

### Why Private Subnets for ECS Tasks?
- **Security**: TeamCity has no direct internet exposure
- **Compliance**: Meets requirements for production CI/CD systems
- **Controlled egress**: All outbound traffic via NAT Gateway (auditable)

**Alternative Considered**: Public subnets
**Why Not Used**: Increases attack surface, violates least-privilege principle

---

4. Add Cost Estimation Section (Before "Getting Started")

## Cost Considerations

⚠️ **TeamCity infrastructure costs vary based on usage. Typical range: $300-600/month (plus TeamCity licensing).**

### Cost Breakdown (us-east-1)

| Component | Configuration | Typical Cost/Month | Notes |
|-----------|---------------|------------|-------|
| **ECS Fargate Server** | 2 vCPU, 4GB RAM, 24/7 | ~$60 | TeamCity server |
| **ECS Fargate Agents** | 4 vCPU, 8GB RAM, variable | ~$100-300 | Build agents (usage-based) |
| **RDS Database** | db.t3.medium, 100GB storage | ~$100 | PostgreSQL for TeamCity |
| **Application Load Balancer** | 1 ALB, 24/7 | ~$20 | HTTPS termination |
| **EFS** | 100GB storage | ~$30 | TeamCity data directory |
| **Data Transfer** | 100GB outbound | ~$9 | Artifact downloads |
| **CloudWatch Logs** | 10GB ingested, 30-day retention | ~$5 | Log storage |
| **TeamCity License** | Variable | **$299+/month** | Per-server or per-agent licensing |

**Typical Monthly Range**: 
- **Infrastructure only**: ~$324-624/month
- **Infrastructure + License**: ~$623-923/month

**Important**: TeamCity licensing costs can exceed infrastructure costs. Review [JetBrains pricing](https://www.jetbrains.com/teamcity/buy/) for accurate licensing estimates.

### Cost Optimization

1. **Stop TeamCity outside business hours**:
   \`\`\`bash
   # Stop TeamCity (scale to 0)
   aws ecs update-service --cluster teamcity --service teamcity-server --desired-count 0
   \`\`\`
   **Potential Savings**: ~$40/month if stopped 16 hours/day on weekdays

2. **Use Spot Instances for build agents**:
   - ECS Fargate Spot provides up to 70% discount for interruptible workloads
   - Suitable for builds that can tolerate interruptions
   **Potential Savings**: Variable based on agent usage

3. **Reduce EFS throughput mode**:
   - Use "Bursting" mode instead of "Provisioned" for low-activity TeamCity
   **Potential Savings**: ~$20/month

4. **Right-size RDS instance**:
   - Monitor database utilization and downsize if CPU/memory is consistently low
   **Potential Savings**: ~$50/month (t3.medium → t3.small)

**Use [AWS Pricing Calculator](https://calculator.aws) for accurate estimates based on your specific usage patterns**.

---

5. Add "Production Considerations" Section (Before "Getting Started")

## Production Considerations

When preparing to deploy this module in a production environment, consider the following:

### Security
- Review and restrict \`allowed_cidr_blocks\` to specific IP ranges (avoid 0.0.0.0/0)
- Enable MFA for AWS IAM users with access to TeamCity infrastructure
- Configure VPC Flow Logs for network traffic auditing
- Implement secret rotation policies for database credentials
- Enable CloudTrail for API activity logging
- Review TeamCity user permissions and authentication settings

### High Availability & Reliability
- Deploy RDS in Multi-AZ configuration for automatic failover
- Enable automated RDS backups with appropriate retention period
- Configure auto-scaling policies for build agents
- Enable EFS backups for TeamCity data directory
- Test disaster recovery procedures (restoring from RDS and EFS backups)
- Document runbooks for common failure scenarios

### Monitoring & Observability
- Set up CloudWatch alarms for critical metrics (CPU, memory, disk usage, RDS connections)
- Configure billing alerts for unexpected cost increases
- Implement centralized log aggregation
- Monitor build queue depth and agent utilization
- Set up alerting for failed builds and infrastructure issues
- Track RDS performance metrics (connections, CPU, IOPS)

### Performance
- Right-size ECS task configurations based on actual workload metrics
- Monitor RDS performance and upgrade instance class if needed
- Review EFS throughput and adjust mode if needed (Bursting vs Provisioned)
- Optimize TeamCity build configurations for parallel execution
- Monitor and tune build agent pool sizes

### Licensing & Compliance
- Ensure TeamCity license covers your agent count
- Plan for license renewal and capacity growth
- Document TeamCity version and upgrade schedule
- Review JetBrains support agreements

### Operations
- Document procedures for common operations (adding users, configuring build agents, upgrades)
- Establish backup and retention policies for build artifacts
- Plan for TeamCity version upgrades
- Define incident response procedures
- Train operations team on AWS Console and TeamCity administration
- Establish SLA for build completion times

[gabebatista]

📋 README Structure Template

To maintain consistency across all module READMEs, we're working towards a standardized structure. While we want to stick as closely to this template as possible, it's okay to make changes to fit the specific module's needs.

Template Structure:

```markdown

[Module/Service Name]

One-sentence description: Brief, clear statement of what this module does and its primary purpose.

Table of Contents

• Overview
• Why This Module?
• Architecture
• Cost Considerations
• Production Considerations
• Quick Start
• Usage Examples
• Configuration Reference
• Operations
• Troubleshooting

Overview

Brief 2-3 paragraph introduction covering:

• What this module deploys (high-level components)
• Primary use case(s)
• Target audience (game studios, indie developers, enterprise teams, etc.)

Why This Module?

Note: For modules with submodules (e.g., Perforce with P4 Server, P4 Auth, P4 Code Review), add a "Module Components" section here explaining what each component does and when to deploy them.

Problem Statement

What problem does this solve for game development teams?

Why [This Tool/Service]?

• vs Alternative 1: Key differentiator (e.g., "Perforce vs Git: Binary file handling")
• vs Alternative 2: Key differentiator (e.g., "Self-hosted vs SaaS: Control and compliance")
• vs Alternative 3: Key differentiator (e.g., "TeamCity vs Jenkins: UI polish vs flexibility")
• Best for: Team size, project type, specific requirements

Note: For modules with multiple alternatives (e.g., CI/CD options), include comparison tables in this section:

Factor	This Module	Alternative Module
Factor 1	Advantage	Trade-off
Factor 2	Trade-off	Advantage

When NOT to Use This Module

• Small teams (< X people) - simpler alternatives may suffice
• Text-only repositories - Git may be more cost-effective
• Teams without [specific requirement] - overhead not justified

Architecture

High-Level Architecture

\`\`\`
[Include diagram - keep it simple and focused on data flow]
\`\`\`

Key Components:

• Component 1: One-line purpose
• Component 2: One-line purpose
• Component 3: One-line purpose

Architecture Decisions

Why these choices matter: Understanding the rationale helps you customize safely and explain to stakeholders.

Use collapsible

Details

sections to keep README digestible:

Decision 1: Example Decision

Choice: What was chosen

Why:

• Reason 1
• Reason 2
• Reason 3

Trade-off:

• What you give up with this choice

When to override: When this choice might not fit your needs

Cost Considerations

Estimated Monthly Cost

Small Team (5-10 developers)

Component	Configuration	Cost
Compute	2x t3.medium (24/7)	$60
Storage	500GB EBS	$50
Network	NLB + 1TB transfer	$40
Total		~$150/month

Cost Optimization:

• Schedule dev environments to stop nights/weekends
• Use Savings Plans for steady-state workloads
• Right-size after 2 weeks based on metrics

Use AWS Pricing Calculator for accurate estimates.

Production Considerations

Note: This section provides guidance on things to consider when preparing for production deployment. It's ultimately up to you to determine what's appropriate for your specific environment and requirements.

When preparing to deploy this module in a production environment, consider the following:

Security

• Review and restrict access controls (security groups, CIDR blocks)
• Enable MFA for AWS IAM users with infrastructure access
• Configure VPC Flow Logs for network traffic auditing
• Implement secret rotation policies where applicable
• Enable CloudTrail for API activity logging
• Review IAM permissions for least privilege access

High Availability & Reliability

• Deploy across multiple Availability Zones where supported
• Enable automated backups with appropriate retention periods
• Test disaster recovery procedures (restoring from backups)
• Document runbooks for common failure scenarios

Monitoring & Observability

• Set up CloudWatch alarms for critical metrics
• Configure billing alerts for unexpected cost increases
• Implement centralized log aggregation
• Define and monitor relevant SLAs
• Set up alerting for service degradation or failures

Performance

• Right-size resources based on actual workload metrics
• Monitor performance metrics and adjust configurations as needed
• Plan for capacity growth as team/usage scales

Operations

• Document procedures for common operations
• Establish backup and retention policies
• Plan for version upgrades and patching
• Define incident response procedures
• Train operations team on troubleshooting and maintenance
  ```

Target Length

500-800 lines (excluding terraform-docs) to keep documentation digestible.