Improve codebase organization for AI-assisted development

.github/workflows/python-pulumi.yml

@@ -4,9 +4,9 @@ on:
   push:
     branches: [main]
     tags: [v*]
-    paths: [python-pulumi/**,.github/workflows/python-pulumi.yml]
+    paths: [python-pulumi/**,lib/types/**,scripts/validate-config-sync.py,.github/workflows/python-pulumi.yml]
   pull_request:
-    paths: [python-pulumi/**,.github/workflows/python-pulumi.yml]
+    paths: [python-pulumi/**,lib/types/**,scripts/validate-config-sync.py,.github/workflows/python-pulumi.yml]
 
 permissions:
   actions: read
@@ -65,3 +65,19 @@ jobs:
 
       - run: just lint
         working-directory: python-pulumi
+
+  validate-config-sync:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: extractions/setup-just@v2
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Validate Go↔Python config sync
+        run: just validate-config-sync

CLAUDE.md

@@ -173,3 +173,114 @@ When contributing to the project:
 # Additional Instructions
 - LLM coding instructions shared with copilot: [.github/copilot/copilot-instructions.md](.github/copilot/copilot-instructions.md)
 - Follow the template in [.github/pull_request_template.md](.github/pull_request_template.md) to format PR descriptions correctly
+
+## Architecture Overview
+
+Brief pointer section:
+- **Config Flow**: How YAML config flows through Go to Python → See `docs/architecture/config-flow.md`
+- **Step Dependencies**: Deployment pipeline ordering and why → See `docs/architecture/step-dependencies.md`
+- **Pulumi Conventions**: Resource naming, Output handling, autoload pattern → See `docs/architecture/pulumi-conventions.md`
+
+## Danger Zones
+
+### Pulumi Resource Names
+- **NEVER** change the first argument (logical name) to a Pulumi resource constructor without understanding state implications
+- Changing `aws.s3.Bucket("my-bucket-name", ...)` to `aws.s3.Bucket("different-name", ...)` causes Pulumi to DELETE the old bucket and CREATE a new one
+- This applies to ALL resources: VPCs, RDS instances, S3 buckets, IAM roles, EKS clusters, etc.
+- If you need to rename a resource, discuss the state migration strategy first
+
+### Config Changes Require Both Languages
+- Adding/modifying a config option requires changes in BOTH:
+  - **Go**: Struct in `lib/types/workload.go` (with YAML struct tags)
+  - **Python**: Dataclass in `python-pulumi/src/ptd/aws_workload.py` or `python-pulumi/src/ptd/__init__.py`
+- Field names must match: Go YAML tags (snake_case) = Python dataclass field names
+- There is no automated validation between the two — mismatches fail at runtime
+
+### Builder Method Ordering
+- `AWSEKSCluster` uses a builder pattern where `with_*()` methods have ordering dependencies
+- Example: `with_node_role()` MUST be called before `with_node_group()` (sets `self.default_node_role`)
+- Check method dependencies before reordering calls
+
+### Resource Naming Conventions
+
+**AWS:**
+- IAM roles: `f"{purpose}.{compound_name}.posit.team"`
+- S3 buckets: `f"{compound_name}-{purpose}"`
+- EKS clusters: `f"default_{compound_name}-control-plane"`
+- All naming methods are on `AWSWorkload` class in `python-pulumi/src/ptd/aws_workload.py`
+
+**Azure:**
+- Resource Groups: `f"rsg-ptd-{sanitized_name}"`
+- Key Vault: `f"kv-ptd-{name[:17]}"` (max 24 chars)
+- Storage Accounts: `f"stptd{name_no_hyphens[:19]}"` (NO hyphens, max 24 chars)
+- VNets: `f"vnet-ptd-{compound_name}"`
+- All naming methods are on `AzureWorkload` class in `python-pulumi/src/ptd/azure_workload.py`
+- Azure tags must use `azure_tag_key_format()` which converts `.` to `/`
+
+Do NOT introduce new naming patterns — follow existing conventions
+
+## Key Patterns
+
+### The autoload Pattern
+Go generates `__main__.py` dynamically (see `lib/pulumi/python.go:WriteMainPy`):
+```python
+import ptd.pulumi_resources.<module>
+ptd.pulumi_resources.<module>.<Class>.autoload()
+```
+- Module: `{cloud}_{target_type}_{step_name}` (e.g., `aws_workload_persistent`)
+- Class: `{Cloud}{TargetType}{StepName}` (e.g., `AWSWorkloadPersistent`)
+- `__main__.py` is NOT in source control — it's generated at runtime
+
+### AWS vs Azure Infrastructure Patterns
+
+**AWS (EKS):**
+- Uses builder pattern with `with_*()` methods
+- Builder methods have ordering dependencies (e.g., `with_node_role()` must come before `with_node_group()`)
+- EKS step is Python-based (`AWSEKSCluster` class)
+
+**Azure (AKS):**
+- AKS step is Go-based (`lib/steps/aks.go`) unlike the Python-based EKS step
+- Azure persistent resources use simple `_define_*()` methods (no builder pattern)
+- No ordering dependencies between `_define_*()` methods
+
+### Pulumi Output[T]
+- Resource properties return `Output[T]`, not plain values
+- Use `.apply(lambda x: ...)` to transform; cannot use in f-strings directly
+- Combine with `pulumi.Output.all(a, b).apply(lambda args: ...)`
+
+### Step Execution
+Steps run sequentially via `ptd ensure`:
+1. `bootstrap` (Go) → 2. `persistent` (Python) → 3. `postgres_config` (Python) → 4. `eks`/`aks` → 5. `clusters` → 6. `helm` → 7. `sites` → 8. `persistent_reprise` (Go)
+
+Each step produces outputs consumed by later steps. See `docs/architecture/step-dependencies.md`.
+
+## Python Pulumi Development
+
+### Testing
+- Use `pulumi.runtime.set_mocks()` for Pulumi resource tests
+- For Go→Python integration details, see the "Go→Python Integration" section above
+- Tests must set `PTD_ROOT` via `monkeypatch.setenv("PTD_ROOT", ...)`
+- See `python-pulumi/tests/` for examples
+- Run: `just test-python-pulumi`
+
+### Adding a New Pulumi Resource Module
+1. Create `python-pulumi/src/ptd/pulumi_resources/<cloud>_<target_type>_<step_name>.py`
+2. Define a class inheriting from `pulumi.ComponentResource`
+3. Implement `@classmethod autoload(cls)` that reads stack name and constructs workload
+4. Add corresponding step in `lib/steps/`
+5. Register step in `WorkloadSteps` or `ControlRoomSteps` in `lib/steps/steps.go`
+
+### Large Files (>1000 lines)
+These files are large and require careful context management:
+
+**AWS:**
+- `pulumi_resources/aws_eks_cluster.py` (~2580 lines) — EKS cluster provisioning with builder pattern
+- `pulumi_resources/aws_workload_persistent.py` (~1454 lines) — VPC, RDS, S3, IAM
+- `pulumi_resources/aws_workload_helm.py` (~1390 lines) — Helm chart deployments (AWS)
+- `__init__.py` (~1275 lines) — Base types, constants, utility functions
+- `aws_workload.py` (~815 lines) — AWS workload config and naming conventions
+
+**Azure:**
+- `pulumi_resources/azure_workload_persistent.py` (~817 lines) — VNet, Postgres, Storage, ACR
+- `pulumi_resources/azure_workload_helm.py` (~675 lines) — Helm chart deployments (Azure)
+- `azure_workload.py` (~398 lines) — Azure workload config and naming with strict char limits

Justfile

@@ -174,6 +174,11 @@ cli:
 check-python-pulumi:
   cd {{ justfile_directory() }}/python-pulumi && just check
 
+# Validate Go↔Python config field sync
+[group('check')]
+validate-config-sync:
+  python3 {{ justfile_directory() }}/scripts/validate-config-sync.py
+
 #############################################################################
 # Format targets
 #############################################################################

docs/README.md

@@ -9,6 +9,12 @@ Welcome to the Posit Team Dedicated (PTD) documentation.
 - [PTD CLI Reference](cli/PTD_CLI_REFERENCE.md) - Complete CLI command documentation
 - [Adding Config Options](guides/adding-config-options.md) - How to add new configuration options to Team Operator
 
+### Architecture
+
+- [Config Flow](architecture/config-flow.md) - How configuration flows from YAML through Go to Python
+- [Step Dependencies](architecture/step-dependencies.md) - The step execution pipeline and dependencies
+- [Pulumi Conventions](architecture/pulumi-conventions.md) - Pulumi-specific patterns and resource naming
+
 ### For Operators
 
 - [Team Operator Overview](team-operator/README.md) - Understanding the Kubernetes operator
@@ -25,6 +31,7 @@ Welcome to the Posit Team Dedicated (PTD) documentation.
 
 ```
 docs/
+├── architecture/           # Architecture documentation
 ├── cli/                    # PTD CLI documentation
 ├── team-operator/          # Team Operator documentation
 ├── guides/                 # How-to guides