Milestone 3 Phase 3.4: Extract server selection algorithm for unit testing

[renecannao]

Parent Issue

Part of #5479 — Unit Testing Milestone 3: Extract Testable Logic from Coupled Classes

Context

When ProxySQL routes a query to a hostgroup, it must select one server from potentially many. The selection considers weights, server status (ONLINE/SHUNNED/OFFLINE), max_connections, max_replication_lag, and max_latency. This algorithm is currently embedded inside `get_connection()` / `get_random_MySrvC()` in the HostGroups Manager, coupled to real connection creation.

Extracting the selection algorithm would enable testing weighted distribution, failover behavior, and health-based exclusion without real network connections.

Current Code

File: `lib/Base_HostGroups_Manager.cpp` (templated)

The server selection function:

• Iterates through servers in a hostgroup
• Filters out servers that are OFFLINE_HARD, SHUNNED, or at max_connections
• Filters by replication lag and latency thresholds
• Selects from remaining candidates using weighted random selection

File: `include/Base_HostGroups_Manager.h`

• `MySrvC` struct contains: address, port, weight, max_connections, status, max_replication_lag, max_latency_us, ConnectionsUsed, ConnectionsFree

Scope

Extract:

`select_server_from_candidates()` — Pure selection algorithm:
```cpp
struct ServerCandidate {
int index; // index into the server list
int64_t weight;
enum MySerStatus status;
unsigned int current_connections;
unsigned int max_connections;
unsigned int current_replication_lag;
unsigned int max_replication_lag;
unsigned int current_latency_us;
unsigned int max_latency_us;
};

/**

• @brief Select a server from a list of candidates using weighted random selection.
• @param candidates Array of server candidates
• @param count Number of candidates
• @param random_seed Seed for deterministic testing
• @return Index of selected server, or -1 if no eligible server found
  /
  int select_server_from_candidates(
  const ServerCandidate candidates,
  int count,
  unsigned int random_seed
  );
  ```

Unit tests should cover:

• Single server → always selected
• Two servers equal weight → roughly 50/50 distribution (statistical test)
• Two servers 3:1 weight → roughly 75/25 distribution
• Server at max_connections → skipped
• OFFLINE_HARD server → never selected
• OFFLINE_SOFT server → not selected for new connections
• SHUNNED server → not selected
• All servers offline → returns -1
• Server exceeding max_replication_lag → excluded
• Server exceeding max_latency → excluded
• Weight=0 server → never selected
• Large number of servers (100+) → distribution matches weights

IMPORTANT: Implementation Instructions

Branch and PR rules

• Create your branch from `v3.0-5473`, NOT from `v3.0`
• Target the PR to `v3.0-5473`, NOT to `v3.0`
• Branch name should be `v3.0-5492`

Where to place files

• Extracted function declarations: create `include/ServerSelection.h` with include guard (follow the pattern of `include/ConnectionPoolDecision.h`)
• Extracted function implementations: place in an appropriate `.cpp` file in `lib/`
• Test file MUST go in `test/tap/tests/unit/server_selection_unit-t.cpp`

How to write the test file

The project has a unit test harness. Your test file MUST:

1. Include the harness headers:
   ```cpp
   #include "tap.h"
   #include "test_globals.h"
   #include "test_init.h"
   ```

2. Include proxysql headers for the types you need:
   ```cpp
   #include "proxysql.h"
   #include "ServerSelection.h" // your new header
   ```

3. Call `test_init_minimal()` at the start of `main()`

4. Call `test_cleanup_minimal()` at the end

5. Use TAP functions: `plan()`, `ok()`, `exit_status()`

DO NOT:

• Do NOT reimplement the extracted functions in the test file — the test must call the REAL functions from `libproxysql.a`
• Do NOT manually define `noise_failures`, `noise_failure_mutex`, `stop_noise_tools`, or `get_noise_tools_count` — these are already provided by `test_globals.cpp`
• Do NOT place the test in `test/tap/tests/` — it MUST go in `test/tap/tests/unit/`
• Do NOT merge `v3.0-5473` into your branch — rebase instead if needed

How to register the test in the Makefile

Edit `test/tap/tests/unit/Makefile`:

1. Add the test name to the `UNIT_TESTS` variable (append to existing list)
2. Add a build rule following the existing pattern:
   ```makefile
   server_selection_unit-t: server_selection_unit-t.cpp $(TEST_HELPERS_OBJ) $(LIBPROXYSQLAR)
   $(CXX) $< $(TEST_HELPERS_OBJ) $(IDIRS) $(LDIRS) $(OPT) \
   $(LIBPROXYSQLAR_FULL) $(STATIC_LIBS) $(MYLIBS) \
   $(ALLOW_MULTI_DEF) -o $@
   ```

Reference examples

Look at these existing files for the correct pattern:

• `test/tap/tests/unit/connection_pool_unit-t.cpp` — unit test using harness
• `test/tap/tests/unit/rule_matching_unit-t.cpp` — unit test using harness
• `include/ConnectionPoolDecision.h` — standalone extracted function header with include guard

Refactoring Challenges

• The current implementation uses `MySrvC` objects directly, which contain connection pool pointers. The extraction must use a lightweight struct (like `ServerCandidate` above) with only the decision-relevant fields.
• GTID-based filtering is MySQL-specific and adds complexity — it can be excluded from the initial extraction.
• The actual `get_random_MySrvC()` also handles GTID constraints — those can be a separate extraction.

Acceptance Criteria

• Extracted function uses a simple struct (no MySrvC dependency)
• Existing E2E TAP tests pass unchanged
• Weighted distribution verified statistically
• All server status types tested
• Edge cases (empty hostgroup, all offline, max_connections) tested
• PR targets `v3.0-5473` (NOT `v3.0`)
• Test file is in `test/tap/tests/unit/`
• Test uses harness (`test_globals.h`, `test_init.h`)
• Test does NOT reimplement extracted functions
• Doxygen documentation on extracted function

[coderabbitai]

🔗 Related PRs

#5344 - Fix wrong index in connection cleanup loops (MySQL and PgSQL) [merged]
#5385 - Fix #2233: Mirror sessions destination_hostgroup overwritten by fast routing [merged]
#5388 - Fix: Extended query Routing and Statistics Tracking [merged]
#5460 - Fix TAP tests for CI migration to per-container hostnames [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[renecannao]

@coderabbitai plan

[coderabbitai]

Coding Plan

Summary

• Create a lightweight ServerCandidate struct that captures only the decision-relevant fields from MySrvC, avoiding heavy connection pool dependencies
• Extract the filtering logic (status, connections, latency, replication lag) and weighted random selection algorithm into a pure function that accepts a seed for deterministic testing
• Follow the established unit test harness pattern from Milestone 3's earlier phases, calling the real extracted functions from libproxysql.a rather than reimplementing logic in tests
• Use statistical tests for weighted distribution validation while maintaining determinism through seeded RNG

Design Choices

Design Choice 1: Earlier milestone phases completed

Options Considered:

1. Create all harness infrastructure as part of this ticket
2. Assume earlier phases created the infrastructure; follow established patterns

Chosen Option: 2

Rationale: The ticket explicitly positions this as "Phase 3.4" and references earlier unit tests as patterns, indicating the infrastructure exists when this phase executes.

Design Choice 2: Deterministic random selection approach

Options Considered:

1. Accept a random_seed parameter and use a local RNG within the function
2. Accept a pre-computed random value directly (the caller seeds and generates)
3. Use a callback/functor for RNG injection

Chosen Option: 1

Rationale: Accept random_seed and use a local seeded RNG. This matches the ticket's function signature and enables statistical distribution tests by providing known seeds.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Create ServerSelection Header

Define the lightweight extraction interface that decouples server selection logic from the heavy MySrvC dependencies.

Task 1: Create ServerCandidate struct definition

Define a minimal struct containing only the fields needed for selection decisions, avoiding any connection pool or network dependencies.

• Create include/ServerSelection.h with standard include guards (#ifndef __CLASS_SERVER_SELECTION_H)
• Define ServerCandidate struct with fields: index (int), weight (int64_t), status (enum MySerStatus), current_connections (unsigned int), max_connections (unsigned int), current_replication_lag (unsigned int), max_replication_lag (unsigned int), current_latency_us (unsigned int), max_latency_us (unsigned int)
• Include only necessary headers: <cstdint> and forward-declare or include MySerStatus enum from proxysql_structs.h
• Add Doxygen documentation block describing the struct's purpose

Task 2: Declare select_server_from_candidates function

Define the pure function signature that performs weighted random selection from filtered candidates.

• Declare int select_server_from_candidates(const ServerCandidate* candidates, int count, unsigned int random_seed) in include/ServerSelection.h
• Add Doxygen documentation with @brief, @param, and @return tags explaining: filtering logic (status, max_connections, latency, replication lag), weighted random selection, and return value (-1 when no eligible server)
• Optionally declare a helper function bool is_candidate_eligible(const ServerCandidate& candidate, unsigned int default_max_latency_ms) if useful for clarity

🤖 Prompt for AI agents

Create the file `include/ServerSelection.h` — a new header that defines the
lightweight server selection interface decoupled from heavy `MySrvC`
dependencies.

- Add standard include guards using `#ifndef __CLASS_SERVER_SELECTION_H`
- Include only `<cstdint>` and the `MySerStatus` enum from `proxysql_structs.h`
- Define the `ServerCandidate` struct with the following fields:
  - `index` (int)
  - `weight` (int64_t)
  - `status` (enum MySerStatus)
  - `current_connections` (unsigned int)
  - `max_connections` (unsigned int)
  - `current_replication_lag` (unsigned int)
  - `max_replication_lag` (unsigned int)
  - `current_latency_us` (unsigned int)
  - `max_latency_us` (unsigned int)
- Add a Doxygen documentation block on the struct describing its purpose
(lightweight, decision-relevant fields only)
- Declare the function: `int select_server_from_candidates(const
ServerCandidate* candidates, int count, unsigned int random_seed)`
- Add Doxygen `@brief`, `@param`, and `@return` tags covering: filtering logic
(status, max_connections, latency, replication lag), weighted random selection
behavior, and that -1 is returned when no eligible server exists
- Optionally declare helper: `bool is_candidate_eligible(const ServerCandidate&
candidate, unsigned int default_max_latency_ms)` if it aids clarity

Phase 2: Implement Server Selection Algorithm

Extract the core filtering and weighted selection logic from get_random_MySrvC() into a standalone implementation.

Task 1: Implement candidate eligibility filtering

Port the multi-gate filter logic from get_random_MySrvC() to work with ServerCandidate structs.

• Create implementation file lib/ServerSelection.cpp
• Include ServerSelection.h and necessary headers
• Implement eligibility check: status must be MYSQL_SERVER_STATUS_ONLINE, current_connections < max_connections, current_latency_us < max_latency_us (or default), current_replication_lag <= max_replication_lag (when max > 0)
• Exclude GTID and Aurora-specific logic per ticket scope
• Exclude SHUNNED recovery logic (that's runtime state management, not selection)

Task 2: Implement weighted random selection

Port the weighted random selection algorithm using a seeded RNG for deterministic testing.

• Compute total weight sum across all eligible candidates
• Use provided random_seed to initialize a local RNG (e.g., simple LCG or seed srand() locally)
• Generate random value in range [1, sum]
• Linear scan candidates accumulating weights until threshold crossed
• Return selected candidate's index field, or -1 if no eligible candidates

Task 3: Integrate with build system

Ensure the new source file is compiled into libproxysql.a.

• Add lib/ServerSelection.cpp to the appropriate target in lib/Makefile (follow pattern of other .cpp files)
• Verify header is accessible from include paths

🤖 Prompt for AI agents

Create `lib/ServerSelection.cpp`, implementing the server selection algorithm
extracted from `get_random_MySrvC()` in `lib/MyHGC.cpp`, and update the build
system to compile it.

**Eligibility filtering** (`is_candidate_eligible` or inline logic in
`select_server_from_candidates`):
- A candidate is eligible only if ALL of the following are true:
  - `status == MYSQL_SERVER_STATUS_ONLINE`
  - `current_connections < max_connections`
- `current_latency_us < max_latency_us` (or falls within a default if
`max_latency_us` is unset)
- `current_replication_lag <= max_replication_lag` — only enforced when
`max_replication_lag > 0`
- Do NOT include GTID logic, Aurora-specific logic, or SHUNNED recovery logic

**Weighted random selection** in `select_server_from_candidates`:
- First pass: compute total weight sum across all eligible candidates
- If no eligible candidates exist, return -1
- Use `random_seed` to initialize a local RNG (e.g., seed `srand()` locally or
use a simple LCG)
- Generate a random value in range `[1, sum]`
- Second pass: linear scan eligible candidates, accumulating weights until the
threshold is crossed
- Return the `index` field of the selected candidate

**Build system:**
- Add `lib/ServerSelection.cpp` to the appropriate target in `lib/Makefile`,
following the same pattern used by other `.cpp` files in that directory
- Confirm the `include/` path is already on the include search path so
`ServerSelection.h` is found

Phase 3: Create Unit Tests

Implement comprehensive unit tests covering all selection scenarios using the established test harness pattern.

Task 1: Set up unit test file structure

Create the test file following the established unit test pattern from earlier milestone phases.

• Create test/tap/tests/unit/server_selection_unit-t.cpp
• Include harness headers: tap.h, test_globals.h, test_init.h
• Include ServerSelection.h to access the real extracted functions
• Set up main() with test_init_minimal() at start and test_cleanup_minimal() before return exit_status()
• Call plan(N) with total test count

Task 2: Implement basic selection tests

Test fundamental selection behavior with simple configurations.

• Test single server (always selected)
• Test server with weight=0 (never selected)
• Test empty candidate list (returns -1)
• Test all servers offline/ineligible (returns -1)
• Use ok() assertions with descriptive messages

Task 3: Implement status filtering tests

Verify correct handling of all MySerStatus values.

• Test MYSQL_SERVER_STATUS_ONLINE → eligible
• Test MYSQL_SERVER_STATUS_OFFLINE_HARD → excluded
• Test MYSQL_SERVER_STATUS_OFFLINE_SOFT → excluded for new connections
• Test MYSQL_SERVER_STATUS_SHUNNED → excluded
• Test MYSQL_SERVER_STATUS_SHUNNED_REPLICATION_LAG → excluded

Task 4: Implement threshold filtering tests

Verify max_connections, latency, and replication lag filtering.

• Test server at current_connections == max_connections → excluded
• Test server below max_connections → eligible
• Test server with current_latency_us > max_latency_us → excluded
• Test server with acceptable latency → eligible
• Test server with current_replication_lag > max_replication_lag → excluded (when max > 0)
• Test server with max_replication_lag = 0 → replication lag check disabled

Task 5: Implement weighted distribution tests

Statistically verify weighted random selection produces expected distributions.

• Test two servers with equal weights: run N iterations (e.g., 10000), verify each selected ~50% (within statistical tolerance, e.g., 45-55%)
• Test two servers with 3:1 weight ratio: verify ~75%/25% distribution
• Test large server count (100+): verify distribution proportional to weights
• Use different seeds to ensure determinism (same seed → same result)

Task 6: Register test in Makefile

Add the test to the unit test build system.

• Edit test/tap/tests/unit/Makefile
• Add server_selection_unit-t to UNIT_TESTS variable
• Add build rule following existing pattern, linking against $(LIBPROXYSQLAR) to access real implementation

🤖 Prompt for AI agents

Create `test/tap/tests/unit/server_selection_unit-t.cpp` — a comprehensive TAP
unit test for the extracted server selection logic — and register it in the unit
test build system.

**File structure:**
- Include `tap.h`, `test_globals.h`, and `test_init.h` (established harness
headers from earlier phases)
- Include `ServerSelection.h` to call the real extracted functions
- `main()` must call `test_init_minimal()` first, then `plan(N)` with the total
test count, and `test_cleanup_minimal()` before `return exit_status()`

**Basic selection tests:**
- Single online server with positive weight → always returns its index
- Server with `weight=0` → never selected
- Empty candidate list (`count=0`) → returns -1
- All candidates offline or otherwise ineligible → returns -1
- Use `ok()` with descriptive messages for all assertions

**Status filtering tests** — one test per status value:
- `MYSQL_SERVER_STATUS_ONLINE` → eligible
- `MYSQL_SERVER_STATUS_OFFLINE_HARD` → excluded
- `MYSQL_SERVER_STATUS_OFFLINE_SOFT` → excluded
- `MYSQL_SERVER_STATUS_SHUNNED` → excluded
- `MYSQL_SERVER_STATUS_SHUNNED_REPLICATION_LAG` → excluded

**Threshold filtering tests:**
- `current_connections == max_connections` → excluded
- `current_connections < max_connections` → eligible
- `current_latency_us > max_latency_us` → excluded
- `current_latency_us < max_latency_us` → eligible
- `current_replication_lag > max_replication_lag` (max > 0) → excluded
- `max_replication_lag = 0` → replication lag check disabled (server eligible
regardless of lag)

**Weighted distribution tests:**
- Two equal-weight servers: run 10,000 iterations, assert each is selected
within 45–55%
- Two servers with 3:1 weight ratio: assert ~75%/25% distribution
- 100+ server pool: assert distribution is proportional to weights
- Determinism check: same seed must produce same result across repeated calls

**Build system:**
- Edit `test/tap/tests/unit/Makefile`
- Add `server_selection_unit-t` to the `UNIT_TESTS` variable
- Add a build rule following the existing pattern, linking against
`$(LIBPROXYSQLAR)`

Research

ProxySQL's server selection logic resides in lib/MyHGC.cpp::get_random_MySrvC(), which filters servers by status (ONLINE/OFFLINE/SHUNNED), max_connections, latency, and replication lag before performing weighted random selection. The MySrvC struct in include/Base_HostGroups_Manager.h contains both decision-relevant fields (weight, status, connections) and heavy dependencies (connection pool pointers). The TAP test framework in test/tap/tap/tap.h provides plan(), ok(), diag(), and exit_status(). Unit tests follow patterns in test/tap/tests/unit-strip_schema_from_query-t.cpp and test/tap/tests/anomaly_detector_unit-t.cpp. The test/tap/tests/unit/ subdirectory and associated harness files (test_globals.h, test_init.h) are being created as part of Milestone 3's earlier phases (3.1-3.3).

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Create the file `include/ServerSelection.h` — a new header that defines the
lightweight server selection interface decoupled from heavy `MySrvC`
dependencies.

- Add standard include guards using `#ifndef __CLASS_SERVER_SELECTION_H`
- Include only `<cstdint>` and the `MySerStatus` enum from `proxysql_structs.h`
- Define the `ServerCandidate` struct with the following fields:
  - `index` (int)
  - `weight` (int64_t)
  - `status` (enum MySerStatus)
  - `current_connections` (unsigned int)
  - `max_connections` (unsigned int)
  - `current_replication_lag` (unsigned int)
  - `max_replication_lag` (unsigned int)
  - `current_latency_us` (unsigned int)
  - `max_latency_us` (unsigned int)
- Add a Doxygen documentation block on the struct describing its purpose
(lightweight, decision-relevant fields only)
- Declare the function: `int select_server_from_candidates(const
ServerCandidate* candidates, int count, unsigned int random_seed)`
- Add Doxygen `@brief`, `@param`, and `@return` tags covering: filtering logic
(status, max_connections, latency, replication lag), weighted random selection
behavior, and that -1 is returned when no eligible server exists
- Optionally declare helper: `bool is_candidate_eligible(const ServerCandidate&
candidate, unsigned int default_max_latency_ms)` if it aids clarity
===============================================================================

Task: 2

Create `lib/ServerSelection.cpp`, implementing the server selection algorithm
extracted from `get_random_MySrvC()` in `lib/MyHGC.cpp`, and update the build
system to compile it.

**Eligibility filtering** (`is_candidate_eligible` or inline logic in
`select_server_from_candidates`):
- A candidate is eligible only if ALL of the following are true:
  - `status == MYSQL_SERVER_STATUS_ONLINE`
  - `current_connections < max_connections`
- `current_latency_us < max_latency_us` (or falls within a default if
`max_latency_us` is unset)
- `current_replication_lag <= max_replication_lag` — only enforced when
`max_replication_lag > 0`
- Do NOT include GTID logic, Aurora-specific logic, or SHUNNED recovery logic

**Weighted random selection** in `select_server_from_candidates`:
- First pass: compute total weight sum across all eligible candidates
- If no eligible candidates exist, return -1
- Use `random_seed` to initialize a local RNG (e.g., seed `srand()` locally or
use a simple LCG)
- Generate a random value in range `[1, sum]`
- Second pass: linear scan eligible candidates, accumulating weights until the
threshold is crossed
- Return the `index` field of the selected candidate

**Build system:**
- Add `lib/ServerSelection.cpp` to the appropriate target in `lib/Makefile`,
following the same pattern used by other `.cpp` files in that directory
- Confirm the `include/` path is already on the include search path so
`ServerSelection.h` is found
===============================================================================

Task: 3

Create `test/tap/tests/unit/server_selection_unit-t.cpp` — a comprehensive TAP
unit test for the extracted server selection logic — and register it in the unit
test build system.

**File structure:**
- Include `tap.h`, `test_globals.h`, and `test_init.h` (established harness
headers from earlier phases)
- Include `ServerSelection.h` to call the real extracted functions
- `main()` must call `test_init_minimal()` first, then `plan(N)` with the total
test count, and `test_cleanup_minimal()` before `return exit_status()`

**Basic selection tests:**
- Single online server with positive weight → always returns its index
- Server with `weight=0` → never selected
- Empty candidate list (`count=0`) → returns -1
- All candidates offline or otherwise ineligible → returns -1
- Use `ok()` with descriptive messages for all assertions

**Status filtering tests** — one test per status value:
- `MYSQL_SERVER_STATUS_ONLINE` → eligible
- `MYSQL_SERVER_STATUS_OFFLINE_HARD` → excluded
- `MYSQL_SERVER_STATUS_OFFLINE_SOFT` → excluded
- `MYSQL_SERVER_STATUS_SHUNNED` → excluded
- `MYSQL_SERVER_STATUS_SHUNNED_REPLICATION_LAG` → excluded

**Threshold filtering tests:**
- `current_connections == max_connections` → excluded
- `current_connections < max_connections` → eligible
- `current_latency_us > max_latency_us` → excluded
- `current_latency_us < max_latency_us` → eligible
- `current_replication_lag > max_replication_lag` (max > 0) → excluded
- `max_replication_lag = 0` → replication lag check disabled (server eligible
regardless of lag)

**Weighted distribution tests:**
- Two equal-weight servers: run 10,000 iterations, assert each is selected
within 45–55%
- Two servers with 3:1 weight ratio: assert ~75%/25% distribution
- 100+ server pool: assert distribution is proportional to weights
- Determinism check: same seed must produce same result across repeated calls

**Build system:**
- Edit `test/tap/tests/unit/Makefile`
- Add `server_selection_unit-t` to the `UNIT_TESTS` variable
- Add a build rule following the existing pattern, linking against
`$(LIBPROXYSQLAR)`

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!