Migrate AI TAP group to Unified CI Infrastructure pattern #5461
+490
−361
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,7 +12,7 @@ | |
|
|
||
| # 1. Determine Required Infras | ||
| INFRAS_TO_CHECK="" | ||
| BASE_GROUP=$(echo "${TAP_GROUP}" | sed -E "s/[-_]g[0-9]+.*//") # Strip -g1, -g2, _g1, _g2 etc. | ||
|
|
||
| if [ -n "${TAP_GROUP}" ]; then | ||
| if [ -f "${WORKSPACE}/test/tap/groups/${TAP_GROUP}/infras.lst" ]; then | ||
|
|
@@ -138,6 +138,21 @@ | |
| python3 "${WORKSPACE}/test/scripts/bin/proxysql-tester.py" | ||
| " | ||
|
|
||
| # Execute group-specific pre-cleanup hook if it exists | ||
| # This runs before the test runner container is removed, allowing cleanup | ||
| # of ProxySQL-specific configuration while admin is still accessible | ||
| if [ -n "${TAP_GROUP}" ]; then | ||
|
Check failure on line 144 in test/infra/control/run-tests-isolated.bash
|
||
| PRE_CLEANUP_HOOK="${WORKSPACE}/test/tap/groups/${TAP_GROUP}/pre-cleanup.bash" | ||
| if [ ! -f "${PRE_CLEANUP_HOOK}" ]; then | ||
|
Check failure on line 146 in test/infra/control/run-tests-isolated.bash
|
||
| PRE_CLEANUP_HOOK="${WORKSPACE}/test/tap/groups/${BASE_GROUP}/pre-cleanup.bash" | ||
| fi | ||
|
|
||
| if [ -f "${PRE_CLEANUP_HOOK}" ]; then | ||
|
Check failure on line 150 in test/infra/control/run-tests-isolated.bash
|
||
| echo ">>> Executing group pre-cleanup hook: ${PRE_CLEANUP_HOOK}" | ||
| "${PRE_CLEANUP_HOOK}" || true # Allow cleanup to fail | ||
| fi | ||
| fi | ||
|
Comment on lines
+141
to
+154
There was a problem hiding this comment. Because this script runs with |
||
|
|
||
| # Clean up only the runner container | ||
| echo ">>> Cleaning up Test Runner container" | ||
| docker rm -f "${TEST_CONTAINER}" >/dev/null 2>&1 || true | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,16 @@ | ||||||||||||
| # AI-g1 Subgroup Environment Configuration | ||||||||||||
| # Inherits from parent ai group - duplicated here for POSIX sh compatibility | ||||||||||||
|
Comment on lines
+1
to
+2
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Verify shell declaration presence in this file.
sed -n '1,3p' test/tap/groups/ai-g1/env.shRepository: sysown/proxysql Length of output: 179 🏁 Script executed: # Find other env.sh files in the project and check how they handle shell declarations
fd 'env\.sh' test/tap -type f | head -10 | xargs -I {} sh -c 'echo "=== {} ===" && head -3 {}'Repository: sysown/proxysql Length of output: 229 🏁 Script executed: # Find other env.sh files in the test/tap directory and check their shell declarations
fd 'env\.sh' test/tap --type f | head -15 | while read file; do echo "=== $file ===" && head -3 "$file"; doneRepository: sysown/proxysql Length of output: 2011 🏁 Script executed: # Search for how these env.sh files are sourced in the codebase
rg -n 'test/tap/groups.*env\.sh|source.*env\.sh|\. .*env\.sh' test/ --type sh --type bash -A 2 | head -50Repository: sysown/proxysql Length of output: 87 🏁 Script executed: # Search for how env.sh files are sourced
rg -n 'test/tap/groups.*env\.sh|source.*env\.sh|\. .*env\.sh' test/ -A 2 | head -60Repository: sysown/proxysql Length of output: 2642 🏁 Script executed: # Check how proxysql-tester.py sources the env files
sed -n '655,680p' test/scripts/bin/proxysql-tester.pyRepository: sysown/proxysql Length of output: 1788 Declare the shell dialect for lint stability. Line 1 triggers ShellCheck SC2148 because the script type is unspecified. Since this file is sourced in Proposed patch+# shellcheck shell=sh
# AI-g1 Subgroup Environment Configuration
# Inherits from parent ai group - duplicated here for POSIX sh compatibility📝 Committable suggestion
Suggested change
🧰 Tools🪛 Shellcheck (0.11.0)[error] 1-1: Tips depend on target shell and yours is unknown. Add a shebang or a 'shell' directive. (SC2148) 🤖 Prompt for AI Agents |
||||||||||||
|
|
||||||||||||
| export DEFAULT_MYSQL_INFRA="infra-mysql84" | ||||||||||||
| export DEFAULT_PGSQL_INFRA="docker-pgsql16-single" | ||||||||||||
|
|
||||||||||||
| export TAP_MCPPORT="${TAP_MCPPORT:-6071}" | ||||||||||||
| export MCP_TARGET_ID="${MCP_TARGET_ID:-tap_mysql_default}" | ||||||||||||
| export MCP_AUTH_PROFILE_ID="${MCP_AUTH_PROFILE_ID:-tap_mysql_auth}" | ||||||||||||
| export MCP_PGSQL_TARGET_ID="${MCP_PGSQL_TARGET_ID:-tap_pgsql_default}" | ||||||||||||
| export MCP_PGSQL_AUTH_PROFILE_ID="${MCP_PGSQL_AUTH_PROFILE_ID:-tap_pgsql_auth}" | ||||||||||||
| export MCP_MYSQL_HOSTGROUP_ID="${MCP_MYSQL_HOSTGROUP_ID:-9100}" | ||||||||||||
| export MCP_PGSQL_HOSTGROUP_ID="${MCP_PGSQL_HOSTGROUP_ID:-9200}" | ||||||||||||
|
|
||||||||||||
| export MYSQL_DATABASE="${MYSQL_DATABASE:-test}" | ||||||||||||
| export PGSQL_DATABASE="${PGSQL_DATABASE:-postgres}" | ||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,80 +1,175 @@ | ||
| # AI TAP Group | ||
|
|
||
| This group runs AI/MCP TAP tests using the **Unified CI Infrastructure** pattern, with isolated Docker network and automated infrastructure management. | ||
|
|
||
| ## Architecture | ||
|
|
||
| The AI group uses the standard `test/infra/` pattern: | ||
|
|
||
| - **MySQL Backend**: `infra-mysql84` (3-node MySQL 8.4 cluster with replication) | ||
| - **PostgreSQL Backend**: `docker-pgsql16-single` (single PostgreSQL 16 instance) | ||
| - **MCP Server**: Configured on port 6071 with MySQL and PostgreSQL targets | ||
| - **Network**: Isolated Docker network `${INFRA_ID}_backend` | ||
|
|
||
| ## Group Structure | ||
|
|
||
| The `ai` group follows the **supergroup/subgroup** pattern: | ||
| - `ai` is the supergroup (defines infrastructure and shared configuration) | ||
| - `ai-g1`, `ai-g2`, etc. are subgroups (define specific test sets in `groups.json`) | ||
|
|
||
| When running with `TAP_GROUP=ai-g1`, the system: | ||
| 1. Looks for `test/tap/groups/ai/infras.lst` (infrastructure requirements) | ||
| 2. Looks for `test/tap/groups/ai/env.sh` (environment configuration) | ||
| 3. Runs only tests tagged with `ai-g1` in `groups.json` | ||
|
|
||
| ## Infrastructure Files | ||
|
|
||
| | File | Purpose | | ||
| |------|---------| | ||
| | `infras.lst` | Lists required infrastructures (`infra-mysql84`, `docker-pgsql16-single`) | | ||
| | `env.sh` | Environment variables for MCP configuration | | ||
| | `mcp-config.sql` | SQL template for MCP setup (variables substituted at runtime) | | ||
| | `cleanup.sql` | SQL template for MCP cleanup after tests | | ||
| | `seed-mysql.sql` | Test data for MySQL (seeded on mysql1, replicated to others) | | ||
| | `seed-pgsql.sql` | Test data for PostgreSQL | | ||
| | `setup-infras.bash` | **Group hook**: Configures MCP and seeds test data after infrastructure is ready | | ||
| | `pre-cleanup.bash` | **Group hook**: Removes MCP configuration before test runner cleanup | | ||
|
|
||
| ## MCP Configuration | ||
|
|
||
| The MCP (Model Context Protocol) is automatically configured with: | ||
|
|
||
| - **MySQL Target**: `tap_mysql_default` (hostgroup 9100) | ||
| - **PostgreSQL Target**: `tap_pgsql_default` (hostgroup 9200) | ||
| - **Port**: 6071 (SSL enabled) | ||
|
|
||
| Test data includes: | ||
| - MySQL: `test.tap_mysql_static_customers`, `test.tap_mysql_static_orders` | ||
| - PostgreSQL: `public.tap_pgsql_static_accounts`, `public.tap_pgsql_static_events` | ||
|
|
||
| ## Usage | ||
|
|
||
| ### Running the Full AI Group | ||
|
|
||
| ```bash | ||
| # Set unique INFRA_ID to avoid conflicts with other runs | ||
| export INFRA_ID="ai-test-$(date +%s)" | ||
| export TAP_GROUP="ai-g1" | ||
| export WORKSPACE=$(pwd) | ||
|
|
||
| # Source common environment | ||
| source test/infra/common/env.sh | ||
|
|
||
| # Start infrastructure (ProxySQL + MySQL + PostgreSQL) | ||
| ./test/infra/control/ensure-infras.bash | ||
|
|
||
| # Run tests | ||
| ./test/infra/control/run-tests-isolated.bash | ||
|
|
||
| # Cleanup | ||
| ./test/infra/control/stop-proxysql-isolated.bash | ||
| ./test/infra/control/destroy-infras.bash | ||
| ``` | ||
|
|
||
| ### Running a Single Test | ||
|
|
||
| ```bash | ||
| export INFRA_ID="ai-single-$(date +%s)" | ||
| export TAP_GROUP="ai-g1" | ||
| export TEST_PY_TAP_INCL="mcp_module-t" | ||
| export WORKSPACE=$(pwd) | ||
|
|
||
| source test/infra/common/env.sh | ||
| ./test/infra/control/ensure-infras.bash | ||
| ./test/infra/control/run-tests-isolated.bash | ||
| ./test/infra/control/stop-proxysql-isolated.bash | ||
| ./test/infra/control/destroy-infras.bash | ||
| ``` | ||
|
|
||
| ### Running Specific MCP Test Suites | ||
|
|
||
| **MCP Static Harvest (MySQL + PostgreSQL):** | ||
| ```bash | ||
| export INFRA_ID="ai-harvest-$(date +%s)" | ||
| export TAP_GROUP="ai-g1" | ||
| export TEST_PY_TAP_INCL="test_mcp_static_harvest-t" | ||
| export WORKSPACE=$(pwd) | ||
|
|
||
| source test/infra/common/env.sh | ||
| ./test/infra/control/ensure-infras.bash | ||
| ./test/infra/control/run-tests-isolated.bash | ||
| ./test/infra/control/stop-proxysql-isolated.bash | ||
| ./test/infra/control/destroy-infras.bash | ||
| ``` | ||
|
|
||
| **MCP Discovery Phase B:** | ||
| ```bash | ||
| export INFRA_ID="ai-discovery-$(date +%s)" | ||
| export TAP_GROUP="ai-g1" | ||
| export TEST_PY_TAP_INCL="test_mcp_llm_discovery_phaseb-t" | ||
| export WORKSPACE=$(pwd) | ||
|
|
||
| source test/infra/common/env.sh | ||
| ./test/infra/control/ensure-infras.bash | ||
| ./test/infra/control/run-tests-isolated.bash | ||
| ./test/infra/control/stop-proxysql-isolated.bash | ||
| ./test/infra/control/destroy-infras.bash | ||
| ``` | ||
|
|
||
| **Claude Headless Flow:** | ||
| ```bash | ||
| export INFRA_ID="ai-claude-$(date +%s)" | ||
| export TAP_GROUP="ai-g1" | ||
| export TEST_PY_TAP_INCL="test_mcp_claude_headless_flow-t" | ||
| export WORKSPACE=$(pwd) | ||
|
|
||
| source test/infra/common/env.sh | ||
| ./test/infra/control/ensure-infras.bash | ||
| ./test/infra/control/run-tests-isolated.bash | ||
| ./test/infra/control/stop-proxysql-isolated.bash | ||
| ./test/infra/control/destroy-infras.bash | ||
coderabbitai[bot] marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| # Optional: Run with real Claude execution | ||
| # export TAP_RUN_REAL_CLAUDE=1 | ||
| # export TAP_CLAUDE_MCP_CONFIG=./scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/mcp_config.json | ||
| ``` | ||
|
|
||
| ## Test Categories | ||
|
|
||
| Tests included in the AI group (defined in `groups.json`): | ||
|
|
||
| - **AI Module Tests**: `ai_validation-t`, `ai_error_handling_edge_cases-t`, `ai_llm_retry_scenarios-t` | ||
| - **Anomaly Detection**: `anomaly_detection-t`, `anomaly_detector_unit-t`, `anomaly_detection_integration-t` | ||
| - **GenAI Tests**: `genai_module-t`, `genai_async-t`, `genai_embedding_rerank-t`, `genai_live_validation-t` | ||
| - **MCP Tests**: `mcp_module-t`, `mcp_query_rules-t`, `mcp_stats_refresh-t`, `mcp_semantic_lifecycle-t`, etc. | ||
| - **NL2SQL Tests**: `nl2sql_integration-t`, `nl2sql_prompt_builder-t`, `nl2sql_model_selection-t`, etc. | ||
| - **Vector Tests**: `vector_features-t`, `vector_db_performance-t` | ||
|
|
||
| ## Environment Variables | ||
|
|
||
| | Variable | Default | Description | | ||
| |----------|---------|-------------| | ||
| | `TAP_MCPPORT` | 6071 | MCP server port | | ||
| | `MCP_TARGET_ID` | tap_mysql_default | MySQL MCP target ID | | ||
| | `MCP_PGSQL_TARGET_ID` | tap_pgsql_default | PostgreSQL MCP target ID | | ||
| | `MCP_MYSQL_HOSTGROUP_ID` | 9100 | MySQL hostgroup for MCP | | ||
| | `MCP_PGSQL_HOSTGROUP_ID` | 9200 | PostgreSQL hostgroup for MCP | | ||
| | `TEST_PY_TAP_INCL` | (see env.sh) | Test filter pattern | | ||
|
|
||
| ## Notes | ||
|
|
||
| - All containers run on an isolated Docker network (`${INFRA_ID}_backend`) | ||
| - MySQL replication automatically propagates seed data from mysql1 to mysql2/mysql3 | ||
| - MCP configuration is applied by the group-specific `setup-infras.bash` hook and cleaned up by `pre-cleanup.bash` | ||
| - The `INFRA_ID` must be unique for each test run to avoid conflicts | ||
|
|
||
| ## Group-Specific Hooks | ||
|
|
||
| The AI group uses the generic hook system provided by the Unified CI Infrastructure: | ||
|
|
||
| | Hook | Executed By | Purpose | | ||
| |------|-------------|---------| | ||
| | `setup-infras.bash` | `ensure-infras.bash` after backends start | Configures MCP, seeds test data | | ||
| | `pre-cleanup.bash` | `run-tests-isolated.bash` before cleanup | Removes MCP configuration | | ||
|
|
||
| These hooks are group-specific and live in the `test/tap/groups/ai/` directory. They are discovered and executed automatically by the infrastructure scripts based on `TAP_GROUP`. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| -- AI Group MCP Cleanup | ||
| -- Removes MCP configuration after tests complete | ||
| -- Variables are substituted using envsubst before execution | ||
|
|
||
| DELETE FROM mcp_target_profiles WHERE target_id IN ('${MCP_TARGET_ID}', '${MCP_PGSQL_TARGET_ID}'); | ||
| DELETE FROM mcp_auth_profiles WHERE auth_profile_id IN ('${MCP_AUTH_PROFILE_ID}', '${MCP_PGSQL_AUTH_PROFILE_ID}'); | ||
| LOAD MCP PROFILES TO RUNTIME; | ||
| SAVE MCP PROFILES TO DISK; | ||
|
|
||
| DELETE FROM mysql_servers WHERE hostgroup_id IN (${MCP_MYSQL_HOSTGROUP_ID}); | ||
| DELETE FROM pgsql_servers WHERE hostgroup_id IN (${MCP_PGSQL_HOSTGROUP_ID}); | ||
| LOAD MYSQL SERVERS TO RUNTIME; | ||
| SAVE MYSQL SERVERS TO DISK; | ||
| LOAD PGSQL SERVERS TO RUNTIME; | ||
| SAVE PGSQL SERVERS TO DISK; |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The script uses the
BASE_GROUPvariable (on line 98) to find a fallback setup hook, but it doesn't define this variable within the script. This relies onBASE_GROUPbeing present in the execution environment, which is risky and inconsistent with other scripts likerun-tests-isolated.bashanddestroy-infras.bashthat derive it internally. To make the script more robust and self-contained, you should deriveBASE_GROUPfromTAP_GROUPbefore it's used. For consistency, it's best to use the samesedcommand used indestroy-infras.bash.