Repository: EDB_Testing - AAP with EnterpriseDB PostgreSQL Multi-Datacenter Last Updated: 2026-03-31 Documentation Version: 1.0
New to this repository? Start here:
- Quick Start Guide ⭐ START HERE - OpenShift/RHEL deployment (15-30 min)
- AAP Containerized Quick Start ⭐ NEW - Multi-DC DR deployment (30-60 min planning)
- Main README - Architecture overview and table of contents
- Deployment Guides - Detailed deployment methods
- DR Testing Guide - Complete testing framework
Quick deployment paths:
- OpenShift (15 min): Quick Start Guide - OpenShift
- RHEL with TPA (20 min): Quick Start Guide - RHEL
- AAP Containerized Growth (30 min): AAP Containerized Quick Start - Growth
- AAP Containerized Enterprise (30 min): AAP Containerized Quick Start - Enterprise
- Local testing (30 min): Quick Start Guide - CRC
Need to perform a DR drill?
- DR Testing Guide - Complete testing framework
- DR Scenarios - 6 documented failure scenarios
Choose your deployment method:
| Platform | Guide | Description |
|---|---|---|
| RHEL / Bare Metal | TPA Deployment ⭐ RECOMMENDED | Automated deployment with Trusted Postgres Architect |
| RHEL Manual | RHEL Manual Install | Traditional VM-based installation |
| OpenShift | OpenShift Manual Install | Operator-based deployment on OpenShift |
| OpenShift (Kustomize) | Database Deployment | GitOps-friendly Kustomize manifests |
| AAP on OpenShift | AAP Deployment | AAP operator with external PostgreSQL |
Specialized Deployment Topics:
- EDB Operator Installation - CloudNativePG operator via OLM
- Cross-Cluster Replication - DC1 → DC2 streaming replication
- AAP OpenShift Manifests - Subscription and AnsibleAutomationPlatform CR
- AAP Deployment Reference ⭐ NEW - Database setup, verification, troubleshooting (Gateway, Controller, Hub, EDA)
- EDB Operator Smoke Test - Validation procedures
Understanding the system:
| Document | Description | Read Time |
|---|---|---|
| Architecture Overview ⭐ COMPREHENSIVE | Complete architecture documentation | 45 min |
| Main README Architecture | High-level overview with diagram | 5 min |
| AAP Containerized Growth DR ⭐ NEW | 3-node multi-DC deployment (cost-optimized) | 25 min |
| AAP Containerized Enterprise DR ⭐ NEW | 8-node multi-DC deployment (production-grade) | 30 min |
| Architecture Validation Report | Validation vs Red Hat AAP 2.6 tested models | 15 min |
| RHEL AAP Architecture | AAP on RHEL with systemd services | 10 min |
| OpenShift AAP Architecture | AAP on OpenShift with operator | 10 min |
Architecture Overview covers:
- Component details (GLB, AAP, PostgreSQL clusters)
- Network connectivity and data flow (writes, reads, backups)
- Replication topology (streaming + WAL archiving)
- Datacenter configurations (DC1 active, DC2 passive)
- Scaling strategies (horizontal, vertical, geographic)
- Backup and restore architecture
AAP Containerized Deployment Models:
Choose based on your requirements:
| Topology | VMs | Best For | RTO | Cost |
|---|---|---|---|---|
| Growth | 16 total (3 AAP/DC) | Small-medium, budget-conscious | < 5 min | Lower |
| Enterprise | 26 total (8 AAP/DC) | Production-critical, high-scale | < 5 min | Higher |
Architecture Decisions:
- Active-Passive topology (DC1 primary, DC2 standby)
- Physical streaming replication + WAL archiving to S3
- CloudNativePG operator (OpenShift) or EDB Postgres Advanced (RHEL)
- EDB Failover Manager (EFM) for automated database failover
- Global Load Balancer for traffic management and health-based routing
DR Planning and Testing:
| Document | Purpose | Read Time |
|---|---|---|
| DR Scenarios | 6 documented failure scenarios | 15 min |
| DR Testing Guide | Complete testing framework (10,000+ words) | 45 min |
| DR Testing Implementation Summary | Implementation details and metrics | 10 min |
| Split-Brain Prevention | Database role validation and fencing | 15 min |
| EDB Failover Manager | EFM integration and configuration | 20 min |
DR Validation Reports:
- DR Replication Validation - Architecture assessment (Score: 7.1/10)
- DR Replication Implementation Status - Gap tracking
- Component Testing Results - Script validation on macOS/CRC
- AAP Deployment Validation (CRC) - Local OpenShift testing
DR Scripts:
- See Operational Scripts section below
Day-to-day operations:
- Operations Runbook - AAP cluster management procedures
- AAP Deployment Reference ⭐ NEW - Deployment verification, troubleshooting, scaling
- Script Reference - All automation scripts documented
- Troubleshooting Guide - Common issues and diagnostics
- EDB Failover Manager - EFM integration and VIP management
Key Operational Tasks:
- Scaling AAP up/down: See scale-aap-up.sh, scale-aap-down.sh
- Monitoring replication: See monitor-efm-scripts.sh
- DR failover: See efm-orchestrated-failover.sh
- Data validation: See validate-aap-data.sh
All scripts located in /scripts/:
| Script | Purpose | Usage |
|---|---|---|
| scale-aap-up.sh | Scale AAP to operational state | ./scale-aap-up.sh <dc1|dc2> |
| scale-aap-down.sh | Scale AAP to zero (DR prep) | ./scale-aap-down.sh <dc1|dc2> |
| efm-orchestrated-failover.sh | Full DR failover orchestration | Called by EFM (post-promotion) |
| efm-aap-failover-wrapper.sh | EFM integration hook | Called by EFM with failover context |
| monitor-efm-scripts.sh | Monitor EFM failover events | ./monitor-efm-scripts.sh (CronJob) |
| dr-failover-test.sh | Automated DR testing framework | See DR Testing Guide |
| validate-aap-data.sh | AAP data integrity validation | ./validate-aap-data.sh <dc1|dc2> |
| measure-rto-rpo.sh | RTO/RPO measurement with milestones | ./measure-rto-rpo.sh start <test-id> |
| generate-dr-report.sh | DR test report generation | ./generate-dr-report.sh <test-id> |
Script Documentation:
- Scripts README ⭐ - Quick reference for all scripts
- Scripts Guide - Comprehensive usage guide
- Scripts Library Reference - Shared library functions API
- Scripts Hooks and CI/CD - Pre-commit hooks and quality automation
- Manual Scripts Doc - Operations runbook
Contributing and automation:
- CI/CD Pipeline - GitHub Actions workflows (6,500 words)
- Scripts Hooks and CI/CD ⭐ NEW - Pre-commit hooks, CI checks, and quality automation
- Pre-commit Hooks - Local validation before commit
- CONTRIBUTING.md - Coming soon (see Documentation Audit)
GitHub Actions Workflows:
.github/workflows/yaml-validation.yml- Kubernetes manifest validation.github/workflows/shell-script-testing.yml- Bash script testing.github/workflows/pr-validation.yml- PR validation and security scanning
Testing:
- Component Testing Results - Script validation (macOS/CRC)
- AAP Deployment Validation - End-to-end validation
- run-ci-checks-locally.sh - Run CI checks before pushing
Visibility and alerting:
- DR Testing Guide - Monitoring Section - CronJob-based DR testing
- EDB Operator Metrics - Prometheus ServiceMonitor
- Split-Brain Prevention - Database role monitoring
Planned Documentation:
- Monitoring and Alerting Guide (see Documentation Audit)
- Grafana Dashboard Setup
- PagerDuty Integration
Security considerations:
- Pre-commit Secret Detection -
detect-secretsintegration - RBAC Configuration - DR testing ServiceAccount
- EFM Security - EFM permissions and VIP management
Planned Documentation:
- Security Hardening Guide (see Documentation Audit)
- TLS/SSL Configuration
- Secrets Management (Vault, Sealed Secrets)
Additional resources:
- Documentation Audit Report - Comprehensive documentation assessment
- Glossary - Coming soon - Terminology and abbreviations
- FAQ - Coming soon - Frequently asked questions
- LICENSE - Copyright and licensing
External Links:
Recommended Path:
- TPA Deployment Guide - Automated deployment ⭐
- RHEL AAP Architecture - Reference architecture
- EDB Failover Manager - EFM setup
- Operations Runbook - Day-to-day operations
Alternative:
- RHEL Manual Install - Manual installation
Recommended Path:
- Database Deployment (Kustomize) - Deploy PostgreSQL ⭐
- AAP Deployment - Deploy AAP with external database
- OpenShift AAP Architecture - Reference architecture
- Cross-Cluster Replication - Setup DC1 → DC2 replication
- DR Testing Guide - Test failover procedures
Alternative:
- OpenShift Manual Install - Step-by-step manual deployment
Validation:
Essential Reading:
- DR Scenarios - Understand failure modes
- Operations Runbook - Day-to-day procedures
- DR Testing Guide - Quarterly drill procedures
- Troubleshooting Guide - Issue resolution
- Scripts README - Automation tools
Essential Reading:
- Install TPA - Automated PostgreSQL deployment
- EDB Operator - CloudNativePG operator
- Cross-Cluster Replication - Replication setup
- EDB Failover Manager - EFM integration
- Split-Brain Prevention - Database safety
Essential Reading:
- Main README - Architecture overview
- OpenShift AAP Architecture - Platform design
- CI/CD Pipeline - Automation workflows
- Database Deployment - Kustomize manifests
- DR Testing Guide - Testing framework
Essential Reading:
- AAP Deployment Reference - Deployment verification and troubleshooting
- AAP Deployment - AAP usage and integration
- Red Hat AAP Documentation - Component capabilities and features
- Troubleshooting Guide - Common issues
- Main README - System architecture
| Status | Count | Description |
|---|---|---|
| ✅ Complete | 23 | Comprehensive, tested, up-to-date |
| 4 | Exists but needs expansion (security, monitoring) | |
| ❌ Planned | 3 | Identified in audit, not yet created (GLOSSARY, FAQ, Migration Guide) |
Recent Additions (2026-03-31 to 2026-04-03):
- ✅ DR Testing Guide (10,000+ words)
- ✅ DR Testing Implementation Summary
- ✅ Component Testing Results
- ✅ Split-Brain Prevention Documentation
- ✅ CI/CD Pipeline Documentation
- ✅ Documentation Audit Report
- ✅ Documentation Index (this file)
- ✅ Contributing Guide (CONTRIBUTING.md)
- ✅ Scripts Library Reference (2026-04-03)
- ✅ Scripts Hooks and CI/CD Guide (2026-04-03)
- ✅ Scripts README reorganization (2026-04-03)
- ✅ AAP Deployment Reference (2026-04-03) - Deployment-specific configuration and troubleshooting
Next Documentation Priorities:
- Security Hardening Guide (Week 2)
- Monitoring and Alerting Guide (Week 3)
- Backup and Restore Guide (Week 4)
- GLOSSARY.md (Month 2)
See Documentation Audit Report for complete roadmap.
For questions or issues:
- See Troubleshooting Guide
- Check FAQ (coming soon)
- Review GitHub Issues (if applicable)
For contributions:
- See CONTRIBUTING.md (coming soon)
- Review CI/CD Pipeline for testing requirements
- Ensure pre-commit hooks pass:
pre-commit run --all-files
Documentation feedback welcome! Please:
- Open an issue for corrections or suggestions
- Submit a PR for improvements
- Contact SRE team for urgent documentation needs
Last Documentation Review: 2026-03-31 Next Review: 2026-06-30 (quarterly)
This index is maintained by the SRE team. Auto-generated documentation should reference this index for consistency.