Skip to content

Ci with eloqstore#209

Merged
githubzilla merged 26 commits intoeloqdata:eloq-10.6.10from
githubzilla:ci_with_eloqstore
Jan 16, 2026
Merged

Ci with eloqstore#209
githubzilla merged 26 commits intoeloqdata:eloq-10.6.10from
githubzilla:ci_with_eloqstore

Conversation

@githubzilla
Copy link
Collaborator

@githubzilla githubzilla commented Jan 16, 2026

Summary by CodeRabbit

  • Chores

    • Migrated storage backend to a new eloq store across build, bootstrap and test environments; standardized cloud credential and endpoint keys, added object-store URL handling and eloq store tuning options.
    • Updated tracked submodule reference.
    • Made threadpool size system variable read-only.
  • Refactor

    • Improved system handler startup/shutdown with safer initialization, shutdown and logging.
  • Tests

    • Re-enabled two previously disabled tests.

✏️ Tip: You can customize this high-level summary in your review settings.

@coderabbitai
Copy link

coderabbitai bot commented Jan 16, 2026

Walkthrough

Replaces RocksDB cloud S3 settings with new eloq_store keys across build scripts, config templates, bootstrap/test configs, and data_substrate pointer; adjusts txlog object-store URL composition and switches dss_server builds to ELOQDSS_ELOQSTORE; adds guarded thread init/shutdown handling in the Eloq system handler.

Changes

Cohort / File(s) Summary
Build & CI scripts
concourse/scripts/main.ent.bash, concourse/scripts/pr.ent.bash
Switch -DWITH_DATA_STORE from ROCKSDB_CLOUD_S3 to ELOQDSS_ELOQSTORE; update config substitutions to eloq_store_*; adjust txlog object-store URL composition to use MINIO_ENDPOINT_URL and bucket-based paths.
Server config template
concourse/scripts/dss_server.ini
Remove rocksdb_cloud_* and old AWS credential lines; add eloq_store_* settings (open files limit, cloud endpoint, store path, access/secret keys).
Bootstrap / mtr configs
concourse/scripts/mtr_bootstrap.cnf, concourse/scripts/mtr_bootstrap_ds.cnf, concourse/scripts/mtr_multi_bootstrap.cnf, concourse/scripts/mtr_multi_bootstrap_ds.cnf
Replace S3 endpoint/bucket/prefix with txlog_rocksdb_cloud_object_store_service_url; introduce eloq_store_* keys and update credential placements; minor newline/EOF tweaks.
Test data_substrate configs
storage/eloq/mysql-test/mono_basic/data_substrate.cnf, storage/eloq/mysql-test/mono_main/data_substrate.cnf, storage/eloq/mysql-test/mono_multi/...data_substrate*.cnf
Remove legacy rocksdb_cloud_*, dynamodb_*, bigtable_*; add eloq_store_* settings and eloq DSS placeholders; standardize MinIO credentials and replace S3-style entries with object-store URLs.
Runtime code
storage/eloq/eloq_system_handler.cpp
Make my_thread_init() conditional with error handling that sets shutdown and notifies CV; add DLOG on reload submit; ensure safe join of system handler thread during shutdown.
Tests enabled
storage/eloq/mysql-test/mono_main/disabled.def
Re-enabled two previously commented tests: group_min_max, mdev-21101.
MyCNF formatting
storage/eloq/mysql-test/mono_basic/my.cnf, .../mono_main/my.cnf, .../mono_multi/my.cnf
Normalize eloq_config= assignment whitespace (remove spaces around =); restore a MONO1_SOCK line placement.
Submodule pointer
data_substrate
Bump tracked commit from 25dd6a39... to c2b1c261....

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested labels

trigger-ci

Suggested reviewers

  • liunyl
  • lzxddz
  • thweetkomputer

Poem

🐇 I hopped from RocksDB to eloq_store today,
Paths change to MinIO where the txlogs play.
Keys renamed and configs neat,
Threads shut down tidy, logs concise and sweet.
A small hop forward — build and tests hooray!

🚥 Pre-merge checks | ✅ 1 | ❌ 2
❌ Failed checks (1 warning, 1 inconclusive)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 23.08% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The title 'Ci with eloqstore' is vague and does not clearly communicate the main purpose of the changeset, which involves migrating from RocksDB cloud storage to eloqstore as the data store backend across multiple configuration and build files. Consider using a more descriptive title such as 'Migrate data store configuration from RocksDB Cloud to eloqstore' or 'Replace RocksDB Cloud with eloqstore in CI configuration' to better reflect the scope of changes.
✅ Passed checks (1 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

🧹 Recent nitpick comments
sql/sys_vars.cc (2)

4007-4014: Confirm thread_pool_size mutability is intentional.
Line 4011 marks it READ_ONLY while Line 4014 still wires ON_UPDATE; if runtime changes are still expected, drop READ_ONLY, otherwise consider removing ON_UPDATE to avoid dead code.


6248-6255: Confirm ignore_db_dirs read‑only intent.
The help text implies a reset via a blank argument, but Line 6253 marks it READ_ONLY; either allow runtime updates or adjust the docs to match.


📜 Recent review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 82827bf and a73ba76.

📒 Files selected for processing (12)
  • concourse/scripts/dss_server.ini
  • concourse/scripts/mtr_bootstrap_ds.cnf
  • concourse/scripts/mtr_multi_bootstrap_ds.cnf
  • sql/sys_vars.cc
  • storage/eloq/mysql-test/mono_basic/data_substrate.cnf
  • storage/eloq/mysql-test/mono_basic/my.cnf
  • storage/eloq/mysql-test/mono_main/data_substrate.cnf
  • storage/eloq/mysql-test/mono_main/my.cnf
  • storage/eloq/mysql-test/mono_multi/data_substrate1.cnf
  • storage/eloq/mysql-test/mono_multi/data_substrate2.cnf
  • storage/eloq/mysql-test/mono_multi/data_substrate3.cnf
  • storage/eloq/mysql-test/mono_multi/my.cnf
✅ Files skipped from review due to trivial changes (1)
  • storage/eloq/mysql-test/mono_main/my.cnf
🚧 Files skipped from review as they are similar to previous changes (3)
  • concourse/scripts/dss_server.ini
  • storage/eloq/mysql-test/mono_multi/data_substrate1.cnf
  • storage/eloq/mysql-test/mono_multi/data_substrate3.cnf
🧰 Additional context used
🧠 Learnings (4)
📓 Common learnings
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 154
File: storage/eloq/ha_eloq.cc:2571-2623
Timestamp: 2025-11-07T07:10:40.633Z
Learning: Repo: eloqdata/eloqsql PR: 154
File/Area: storage/eloq/ha_eloq.cc (eloq_init_func cluster membership resolution)
Learning: In this repo’s ng_config, the local node is guaranteed to be a candidate (NodeConfig::is_candidate_ == true). Therefore, assigning native_ng_id only when is_candidate_ is true is acceptable; the key improvement is to break early from both loops once the local node is found.
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 154
File: storage/eloq/ha_eloq.cc:2770-2781
Timestamp: 2025-11-07T05:12:43.208Z
Learning: In eloqdata/eloqsql, storage/eloq/ha_eloq.cc (KV_ELOQDS path): for DATA_STORE_TYPE_ELOQDSS_ROCKSDB (local RocksDB), each eloqsql instance owns its own RocksDB instance, and passing create_if_missing=true to DataStoreService::StartService is intentional and safe.
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 127
File: storage/eloq/build_log_service.cmake:97-124
Timestamp: 2025-09-25T11:41:40.111Z
Learning: The `storage/eloq/build_log_service.cmake` file is specifically for the open log service build and only supports ROCKSDB as its log state backend. Unlike the enterprise version in `build_eloq_log_service.cmake`, it does not need to handle cloud-specific variants like ROCKSDB_CLOUD_S3 or ROCKSDB_CLOUD_GCS.
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 113
File: storage/eloq/ha_eloq.cc:2602-2604
Timestamp: 2025-09-22T04:07:38.428Z
Learning: In EloqDB's RocksDB Cloud integration, the branch_name parameter is used internally by RocksDB Cloud for managing different branches in cloud storage (S3/GCS), not as part of the local file system paths or S3 URL paths. The local data directory structure remains the same regardless of the branch being used.
📚 Learning: 2025-09-25T11:41:40.111Z
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 127
File: storage/eloq/build_log_service.cmake:97-124
Timestamp: 2025-09-25T11:41:40.111Z
Learning: The `storage/eloq/build_log_service.cmake` file is specifically for the open log service build and only supports ROCKSDB as its log state backend. Unlike the enterprise version in `build_eloq_log_service.cmake`, it does not need to handle cloud-specific variants like ROCKSDB_CLOUD_S3 or ROCKSDB_CLOUD_GCS.

Applied to files:

  • storage/eloq/mysql-test/mono_main/data_substrate.cnf
  • storage/eloq/mysql-test/mono_basic/data_substrate.cnf
  • storage/eloq/mysql-test/mono_multi/data_substrate2.cnf
  • concourse/scripts/mtr_bootstrap_ds.cnf
  • concourse/scripts/mtr_multi_bootstrap_ds.cnf
📚 Learning: 2025-11-07T05:12:43.208Z
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 154
File: storage/eloq/ha_eloq.cc:2770-2781
Timestamp: 2025-11-07T05:12:43.208Z
Learning: In eloqdata/eloqsql, storage/eloq/ha_eloq.cc (KV_ELOQDS path): for DATA_STORE_TYPE_ELOQDSS_ROCKSDB (local RocksDB), each eloqsql instance owns its own RocksDB instance, and passing create_if_missing=true to DataStoreService::StartService is intentional and safe.

Applied to files:

  • storage/eloq/mysql-test/mono_main/data_substrate.cnf
  • storage/eloq/mysql-test/mono_basic/data_substrate.cnf
  • storage/eloq/mysql-test/mono_multi/data_substrate2.cnf
  • concourse/scripts/mtr_bootstrap_ds.cnf
  • concourse/scripts/mtr_multi_bootstrap_ds.cnf
📚 Learning: 2025-09-25T11:40:48.717Z
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 127
File: concourse/scripts/main.bash:164-165
Timestamp: 2025-09-25T11:40:48.717Z
Learning: The main.bash script in concourse/scripts/main.bash is for open log build, which only supports ROCKSDB as the log state option and does not need cloud-specific variants like ROCKSDB_CLOUD_S3.

Applied to files:

  • storage/eloq/mysql-test/mono_main/data_substrate.cnf
  • storage/eloq/mysql-test/mono_basic/data_substrate.cnf
  • concourse/scripts/mtr_bootstrap_ds.cnf
  • concourse/scripts/mtr_multi_bootstrap_ds.cnf
🧬 Code graph analysis (1)
sql/sys_vars.cc (2)
sql/wsrep_var.cc (10)
  • wsrep_provider_options_check (535-543)
  • wsrep_provider_options_check (535-535)
  • wsrep_cluster_address_check (627-646)
  • wsrep_cluster_address_check (627-627)
  • wsrep_on_check (149-200)
  • wsrep_on_check (149-149)
  • wsrep_on_update (105-147)
  • wsrep_on_update (105-105)
  • wsrep_start_position_check (345-393)
  • wsrep_start_position_check (345-345)
sql/wsrep_sst.cc (12)
  • wsrep_sst_receive_address_check (217-232)
  • wsrep_sst_receive_address_check (217-217)
  • wsrep_sst_receive_address_update (234-238)
  • wsrep_sst_receive_address_update (234-235)
  • wsrep_sst_auth_check (240-243)
  • wsrep_sst_auth_check (240-240)
  • wsrep_sst_auth_update (290-293)
  • wsrep_sst_auth_update (290-290)
  • wsrep_sst_donor_check (300-303)
  • wsrep_sst_donor_check (300-300)
  • wsrep_sst_donor_update (305-308)
  • wsrep_sst_donor_update (305-305)
🔇 Additional comments (37)
storage/eloq/mysql-test/mono_multi/data_substrate2.cnf (2)

8-9: LGTM!

The consolidation of S3 endpoint, bucket name, and prefix into a single txlog_rocksdb_cloud_object_store_service_url is cleaner and aligns with the broader migration to object-store based configurations.


15-18: No changes needed. The empty eloq_dss_config_file_path is consistent across all data_substrate configuration files (data_substrate1.cnf, data_substrate2.cnf, and data_substrate3.cnf), confirming this is an intentional configuration for these test scenarios.

storage/eloq/mysql-test/mono_main/data_substrate.cnf (3)

9-10: LGTM: txlog object-store URL looks aligned with the new format.

No concerns with the region + object-store URL pairing here.


13-14: Confirm which credential keys are actually consumed.

With both legacy aws_* keys and new eloq_store_cloud_* keys present, please verify there’s no ambiguity in config parsing (or add a clarifying comment if both are intentionally required for different components).


16-21: LGTM: eloq_store settings are consistent with the migration.

No issues spotted in these new keys or values.

storage/eloq/mysql-test/mono_multi/my.cnf (1)

22-29: LGTM — config normalization only.

Whitespace normalization around eloq_config is consistent and should be harmless.

storage/eloq/mysql-test/mono_basic/my.cnf (1)

21-28: LGTM — formatting + ENV socket line look correct.

storage/eloq/mysql-test/mono_basic/data_substrate.cnf (1)

9-21: LGTM — object-store URL + eloq_store_* settings look consistent.

concourse/scripts/mtr_bootstrap_ds.cnf (1)

8-24: LGTM — config aligns with the new eloq_store pattern.

concourse/scripts/mtr_multi_bootstrap_ds.cnf (1)

8-19: Configuration is correct for multi-node Data Substrate deployment; no changes needed.

This config file is intentionally configured for multi-node Data Substrate Service (DSS) using eloq_dss_* settings, not single-node eloq_store_* settings. All equivalent multi-node test configs (storage/eloq/mysql-test/mono_multi/data_substrate*.cnf) consistently use the same pattern—DSS peers only, no eloq_store settings. The separate DSS server process (dss_server) manages the distributed store, so the individual node configs reference it via eloq_dss_config_file_path and eloq_dss_peer_node rather than defining eloq_store directly.

Likely an incorrect or invalid review comment.

sql/sys_vars.cc (27)

545-546: Binlog file cache size description update looks good.


1129-1129: Whitespace-only change; no review needed.


2524-2524: Whitespace-only change; no review needed.


2758-2761: “default” optimizer_switch token appended safely.
Appending at the end preserves existing bit positions.


3097-3097: Whitespace-only change; no review needed.


4019-4027: Thread pool stall limit definition looks fine.


4031-4038: Thread pool max threads definition looks fine.


4460-4460: Comment-only change; no review needed.


4966-4972: Extra filename validation is a good safety check.


5309-5314: Error path stays consistent.


5493-5493: Whitespace-only change; no review needed.


5724-5729: lc_messages now aligns with MY_LOCALE defaults.


5757-5764: Verify wsrep_provider_options validation doesn’t block startup configs.
Line 5763 now hooks wsrep_provider_options_check, which rejects updates when WSREP isn’t started; please confirm this still allows config‑file values or gate it to runtime SET. Based on provided snippets, please verify.


5779-5786: Cluster address validation hook looks good.


5833-5837: wsrep_convert_LOCK_to_trx definition looks good.


5886-5892: wsrep_auto_increment_control definition looks good.


5907-5914: wsrep_sst_receive_address now wired to check/update; LGTM.


5916-5922: wsrep_sst_auth update hook looks good.


5924-5929: wsrep_sst_donor update hook looks good.


5931-5935: wsrep_sst_donor_rejects_queries definition looks good.


5937-5943: wsrep_on sysvar wiring looks good.


5945-5952: wsrep_start_position validation hook looks good.


5971-5974: wsrep_certify_nonPK definition looks good.


6089-6106: wsrep SST/replication option updates look fine.


6330-6335: Slow log filter names update looks fine.


6373-6379: Default regex flags list looks fine.


6432-6438: Slow log verbosity names update looks fine.

✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Fix all issues with AI agents
In `@concourse/scripts/main.ent.bash`:
- Around line 103-104: The sed replacement for eloq_store_cloud_access_key
contains an extra space before the equals sign causing the INI key to be written
with a trailing space; update the sed expression that targets
eloq_store_cloud_access_key (the first sed command) to remove the extraneous
space so it produces "eloq_store_cloud_access_key=${ELOQ_AWS_ACCESS_KEY_ID}"
(matching the pattern used for eloq_store_cloud_secret_key) and ensure the
variable ELOQ_AWS_ACCESS_KEY_ID is inserted without added whitespace.

In `@concourse/scripts/mtr_bootstrap.cnf`:
- Line 8: Replace the hard-coded developer-specific absolute path assigned to
eloq_config in mtr_bootstrap.cnf with a repo-relative or environment-driven
value: change the assignment of the variable eloq_config so it builds the path
relative to the repository root (or reads a base directory from an environment
variable like ELOQ_BASE_DIR) and joins it with the config filename
(mtr_bootstrap_ds.cnf) instead of embedding /home/tianxj/.... Update any
bootstrap logic that reads eloq_config to fallback to a sensible default if the
env var is not set.

In `@concourse/scripts/mtr_multi_bootstrap.cnf`:
- Line 8: The eloq_config setting currently contains a developer-specific
absolute path (/home/tianxj/...) which is not portable; update the eloq_config
entry to use either a repository-relative path (e.g., relative to the script
location) or read from an environment variable (e.g., ELOQ_CONFIG) so the
bootstrap can run on other machines; modify the eloq_config assignment in
concourse/scripts/mtr_multi_bootstrap.cnf to reference the chosen repo-relative
location or an env var and document the expected variable if used.

In `@concourse/scripts/pr.ent.bash`:
- Around line 124-125: The first sed replacement adds an extra space after the =
in the eloq_store_cloud_access_key assignment which can introduce trailing
whitespace; update the sed expression for eloq_store_cloud_access_key to remove
that space so it matches the eloq_store_cloud_secret_key pattern (use the same
substitution style and the ELOQ_AWS_ACCESS_KEY_ID variable), ensuring both
replacements produce "key=VALUE" without extra whitespace in dss_server.ini.
- Around line 105-117: The config edits in pr.ent.bash update templates and sed
replacements under eloqsql_src (update_config_template for mtr_bootstrap_ds.cnf
and mtr_multi_bootstrap_ds.cnf and the sed lines that set eloq_config and
hm_bin_path), but later runtime/test steps still reference eloqsql_pr; to fix,
choose one root and make all references consistent—either change these
occurrences of "$WORKSPACE/eloqsql_src/..." to "$WORKSPACE/eloqsql_pr/..." (for
mtr_bootstrap_ds.cnf, mtr_multi_bootstrap_ds.cnf and hm_bin_path) or update the
downstream MTR/bootstrap and dss_server invocation to point at eloqsql_src;
ensure the same variable/root is used for update_config_template, the sed
replacements (eloq_config and hm_bin_path), and any later test/runtime
references so the configs actually used by tests are the ones you edited.

In `@storage/eloq/eloq_system_handler.cpp`:
- Around line 38-46: The current branch returns early when my_thread_init()
fails, leaving shutdown_ false and the internal thread potentially unjoined;
change the failure path in the my_thread_init() check (and the analogous block
at lines 105-115) to set shutdown_ = true and then ensure the worker thread is
joined if joinable (call Shutdown() or thread_.join()/equivalent) before
returning so no further work can be enqueued and the thread is properly cleaned
up; update the failure handling around my_thread_init(), shutdown_, and the
thread join logic so both init failures and normal shutdown follow the same join
semantics.
♻️ Duplicate comments (3)
concourse/scripts/mtr_multi_bootstrap_ds.cnf (1)

15-16: Same MinIO-credential concern applies here.
Avoid hardcoding access keys if env/CI substitution is supported.

storage/eloq/mysql-test/mono_multi/data_substrate3.cnf (1)

15-18: Same credential + empty DSS config path concern applies here.
Please ensure the empty path is valid and credentials aren’t inadvertently reused outside CI.

storage/eloq/mysql-test/mono_multi/data_substrate2.cnf (1)

15-18: Same credential + empty DSS config path concern applies here.
Please ensure the empty path is valid and credentials aren’t inadvertently reused outside CI.

📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 95e9268 and 2b09b9f.

📒 Files selected for processing (14)
  • concourse/scripts/dss_server.ini
  • concourse/scripts/main.ent.bash
  • concourse/scripts/mtr_bootstrap.cnf
  • concourse/scripts/mtr_bootstrap_ds.cnf
  • concourse/scripts/mtr_multi_bootstrap.cnf
  • concourse/scripts/mtr_multi_bootstrap_ds.cnf
  • concourse/scripts/pr.ent.bash
  • data_substrate
  • storage/eloq/eloq_system_handler.cpp
  • storage/eloq/mysql-test/mono_basic/data_substrate.cnf
  • storage/eloq/mysql-test/mono_main/data_substrate.cnf
  • storage/eloq/mysql-test/mono_multi/data_substrate1.cnf
  • storage/eloq/mysql-test/mono_multi/data_substrate2.cnf
  • storage/eloq/mysql-test/mono_multi/data_substrate3.cnf
🧰 Additional context used
🧠 Learnings (6)
📓 Common learnings
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 154
File: storage/eloq/ha_eloq.cc:2571-2623
Timestamp: 2025-11-07T07:10:40.633Z
Learning: Repo: eloqdata/eloqsql PR: 154
File/Area: storage/eloq/ha_eloq.cc (eloq_init_func cluster membership resolution)
Learning: In this repo’s ng_config, the local node is guaranteed to be a candidate (NodeConfig::is_candidate_ == true). Therefore, assigning native_ng_id only when is_candidate_ is true is acceptable; the key improvement is to break early from both loops once the local node is found.
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 127
File: storage/eloq/build_log_service.cmake:97-124
Timestamp: 2025-09-25T11:41:40.111Z
Learning: The `storage/eloq/build_log_service.cmake` file is specifically for the open log service build and only supports ROCKSDB as its log state backend. Unlike the enterprise version in `build_eloq_log_service.cmake`, it does not need to handle cloud-specific variants like ROCKSDB_CLOUD_S3 or ROCKSDB_CLOUD_GCS.
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 154
File: storage/eloq/ha_eloq.cc:2770-2781
Timestamp: 2025-11-07T05:12:43.208Z
Learning: In eloqdata/eloqsql, storage/eloq/ha_eloq.cc (KV_ELOQDS path): for DATA_STORE_TYPE_ELOQDSS_ROCKSDB (local RocksDB), each eloqsql instance owns its own RocksDB instance, and passing create_if_missing=true to DataStoreService::StartService is intentional and safe.
📚 Learning: 2025-11-07T05:12:43.208Z
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 154
File: storage/eloq/ha_eloq.cc:2770-2781
Timestamp: 2025-11-07T05:12:43.208Z
Learning: In eloqdata/eloqsql, storage/eloq/ha_eloq.cc (KV_ELOQDS path): for DATA_STORE_TYPE_ELOQDSS_ROCKSDB (local RocksDB), each eloqsql instance owns its own RocksDB instance, and passing create_if_missing=true to DataStoreService::StartService is intentional and safe.

Applied to files:

  • concourse/scripts/dss_server.ini
  • storage/eloq/mysql-test/mono_multi/data_substrate3.cnf
  • storage/eloq/mysql-test/mono_basic/data_substrate.cnf
  • concourse/scripts/mtr_multi_bootstrap_ds.cnf
  • storage/eloq/mysql-test/mono_main/data_substrate.cnf
  • concourse/scripts/mtr_bootstrap_ds.cnf
  • concourse/scripts/pr.ent.bash
  • storage/eloq/mysql-test/mono_multi/data_substrate1.cnf
  • concourse/scripts/main.ent.bash
  • storage/eloq/mysql-test/mono_multi/data_substrate2.cnf
📚 Learning: 2025-09-25T11:41:40.111Z
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 127
File: storage/eloq/build_log_service.cmake:97-124
Timestamp: 2025-09-25T11:41:40.111Z
Learning: The `storage/eloq/build_log_service.cmake` file is specifically for the open log service build and only supports ROCKSDB as its log state backend. Unlike the enterprise version in `build_eloq_log_service.cmake`, it does not need to handle cloud-specific variants like ROCKSDB_CLOUD_S3 or ROCKSDB_CLOUD_GCS.

Applied to files:

  • concourse/scripts/dss_server.ini
  • storage/eloq/mysql-test/mono_multi/data_substrate3.cnf
  • storage/eloq/mysql-test/mono_basic/data_substrate.cnf
  • concourse/scripts/mtr_multi_bootstrap_ds.cnf
  • storage/eloq/mysql-test/mono_main/data_substrate.cnf
  • concourse/scripts/mtr_bootstrap_ds.cnf
  • concourse/scripts/pr.ent.bash
  • storage/eloq/mysql-test/mono_multi/data_substrate1.cnf
  • concourse/scripts/main.ent.bash
  • storage/eloq/mysql-test/mono_multi/data_substrate2.cnf
📚 Learning: 2025-09-25T11:40:48.717Z
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 127
File: concourse/scripts/main.bash:164-165
Timestamp: 2025-09-25T11:40:48.717Z
Learning: The main.bash script in concourse/scripts/main.bash is for open log build, which only supports ROCKSDB as the log state option and does not need cloud-specific variants like ROCKSDB_CLOUD_S3.

Applied to files:

  • storage/eloq/mysql-test/mono_basic/data_substrate.cnf
  • concourse/scripts/mtr_multi_bootstrap_ds.cnf
  • storage/eloq/mysql-test/mono_main/data_substrate.cnf
  • concourse/scripts/mtr_bootstrap_ds.cnf
  • concourse/scripts/pr.ent.bash
📚 Learning: 2025-09-25T11:33:33.221Z
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 127
File: concourse/scripts/build_tarball_open.bash:136-136
Timestamp: 2025-09-25T11:33:33.221Z
Learning: The open log service (OPEN_LOG_SERVICE=ON) in concourse/scripts/build_tarball_open.bash only supports ROCKSDB as its log state and does not use the WITH_LOG_STATE parameter that was introduced to replace USE_ROCKSDB_LOG_STATE in other parts of the codebase.

Applied to files:

  • concourse/scripts/mtr_bootstrap_ds.cnf
  • concourse/scripts/pr.ent.bash
  • concourse/scripts/main.ent.bash
📚 Learning: 2025-11-07T07:10:40.633Z
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 154
File: storage/eloq/ha_eloq.cc:2571-2623
Timestamp: 2025-11-07T07:10:40.633Z
Learning: Repo: eloqdata/eloqsql PR: 154
File/Area: storage/eloq/ha_eloq.cc (eloq_init_func cluster membership resolution)
Learning: In this repo’s ng_config, the local node is guaranteed to be a candidate (NodeConfig::is_candidate_ == true). Therefore, assigning native_ng_id only when is_candidate_ is true is acceptable; the key improvement is to break early from both loops once the local node is found.

Applied to files:

  • concourse/scripts/mtr_multi_bootstrap.cnf
🧬 Code graph analysis (1)
concourse/scripts/pr.ent.bash (2)
concourse/scripts/main.ent.bash (1)
  • update_config_template (49-69)
concourse/scripts/main.bash (1)
  • update_config_template (44-64)
🔇 Additional comments (15)
data_substrate (1)

1-1: Verify the submodule update against PR objectives.

Confirm that the data_substrate submodule update (from commit 25dd6a39cedb036e8ce849509245769fbcea48c7 to c2b1c26101ba4bace22b14fe348a6d5d454f5d6d) contains expected changes supporting the stated PR goals and doesn't introduce breaking changes. This requires examining the commit details in the tx_service repository to confirm configuration updates, template modifications, and backward compatibility.

storage/eloq/mysql-test/mono_multi/data_substrate1.cnf (2)

9-9: LGTM for the object-store service URL update.
This aligns the txlog config with the new object-store style.


15-18: These are test fixtures with CI-driven substitution already in place.

The hardcoded minioadmin credentials (lines 15–16) are standard MinIO test defaults appropriate for local test config templates. The concourse CI infrastructure already substitutes credentials via environment variables (ELOQ_AWS_ACCESS_KEY_ID, ELOQ_AWS_SECRET_KEY) and populates the empty eloq_dss_config_file_path (line 18) with a resolved path during execution. No changes needed.

concourse/scripts/mtr_multi_bootstrap_ds.cnf (1)

9-9: LGTM for the object-store service URL update.

storage/eloq/mysql-test/mono_multi/data_substrate3.cnf (1)

9-9: LGTM for the object-store service URL update.

storage/eloq/eloq_system_handler.cpp (1)

55-65: Logging additions look good.
The DLOGs provide helpful lifecycle and work-tracing without affecting release builds.

Also applies to: 68-70, 101-101

storage/eloq/mysql-test/mono_multi/data_substrate2.cnf (1)

9-9: LGTM for the object-store service URL update.

storage/eloq/mysql-test/mono_main/data_substrate.cnf (1)

10-21: LGTM for ELOQSTORE config swap.
Keys and endpoints look consistent with the new eloq_store_* layout.

storage/eloq/mysql-test/mono_basic/data_substrate.cnf (1)

10-21: LGTM for ELOQSTORE config swap.
Configuration aligns with the new eloq_store_* keys.

concourse/scripts/dss_server.ini (1)

9-13: LGTM for ELOQSTORE settings.
Config looks consistent with the new eloq_store_* layout.

concourse/scripts/mtr_bootstrap_ds.cnf (1)

8-9: ELOQSTORE config defaults look consistent.

The new txlog object-store URL and eloq_store_* values align with the updated MinIO-based setup.

Also applies to: 15-24

concourse/scripts/main.ent.bash (2)

57-68: Template substitutions align with new eloq_store keys.

The MinIO endpoint, object-store URL, and eloq_store substitutions look consistent with the ELOQSTORE migration.


131-132: ELOQSTORE build flags applied consistently.

Both the main build and dss_server build now target ELOQDSS_ELOQSTORE as intended.

Also applies to: 150-150

concourse/scripts/pr.ent.bash (2)

78-89: Template substitutions updated for eloq_store.

The new MinIO endpoint, object-store URL, and eloq_store substitutions align with the migration.


153-154: ELOQSTORE build flags applied for dss_server.

The CMake switches to ELOQDSS_ELOQSTORE are consistent across the main build and the dss_server build.

Also applies to: 172-172

✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.

thread_pool_dedicated_listener=1

eloq_config= No newline at end of file
eloq_config=/home/tianxj/hive/eloqdata/eloqsql/concourse/scripts/mtr_bootstrap_ds.cnf
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Avoid developer-specific absolute paths.
/home/tianxj/... will break in CI and on other machines. Use a repo‑relative path or an env-configured base path instead.

💡 Possible fix (adjust to your bootstrap resolution rules)
-eloq_config=/home/tianxj/hive/eloqdata/eloqsql/concourse/scripts/mtr_bootstrap_ds.cnf
+eloq_config=concourse/scripts/mtr_bootstrap_ds.cnf
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
eloq_config=/home/tianxj/hive/eloqdata/eloqsql/concourse/scripts/mtr_bootstrap_ds.cnf
eloq_config=concourse/scripts/mtr_bootstrap_ds.cnf
🤖 Prompt for AI Agents
In `@concourse/scripts/mtr_bootstrap.cnf` at line 8, Replace the hard-coded
developer-specific absolute path assigned to eloq_config in mtr_bootstrap.cnf
with a repo-relative or environment-driven value: change the assignment of the
variable eloq_config so it builds the path relative to the repository root (or
reads a base directory from an environment variable like ELOQ_BASE_DIR) and
joins it with the config filename (mtr_bootstrap_ds.cnf) instead of embedding
/home/tianxj/.... Update any bootstrap logic that reads eloq_config to fallback
to a sensible default if the env var is not set.

Comment on lines 105 to +117
# config mtr_bootstrap.cnf
update_config_template "$WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap_ds.cnf"
sed -i "s|eloq_config=.*|eloq_config=$WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap_ds.cnf|g" $WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap.cnf
sed -i "s|hm_bin_path=.*|hm_bin_path=$WORKSPACE/eloqsql_pr/bld/data_substrate/host_manager|g" $WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap_ds.cnf
update_config_template "$WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap_ds.cnf"
sed -i "s|eloq_config=.*|eloq_config=$WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap_ds.cnf|g" $WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap.cnf
sed -i "s|hm_bin_path=.*|hm_bin_path=$WORKSPACE/eloqsql_src/bld/data_substrate/host_manager|g" $WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap_ds.cnf
echo "mtr_bootstrap_ds.cnf"
cat $WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap_ds.cnf
cat $WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap_ds.cnf

#config mtr_multi_bootstrap.cnf
update_config_template "$WORKSPACE/eloqsql_pr/concourse/scripts/mtr_multi_bootstrap_ds.cnf"
sed -i "s|eloq_config=.*|eloq_config=$WORKSPACE/eloqsql_pr/concourse/scripts/mtr_multi_bootstrap_ds.cnf|g" $WORKSPACE/eloqsql_pr/concourse/scripts/mtr_multi_bootstrap.cnf
sed -i "s|hm_bin_path=.*|hm_bin_path=$WORKSPACE/eloqsql_pr/bld/data_substrate/host_manager|g" $WORKSPACE/eloqsql_pr/concourse/scripts/mtr_multi_bootstrap_ds.cnf
update_config_template "$WORKSPACE/eloqsql_src/concourse/scripts/mtr_multi_bootstrap_ds.cnf"
sed -i "s|eloq_config=.*|eloq_config=$WORKSPACE/eloqsql_src/concourse/scripts/mtr_multi_bootstrap_ds.cnf|g" $WORKSPACE/eloqsql_src/concourse/scripts/mtr_multi_bootstrap.cnf
sed -i "s|hm_bin_path=.*|hm_bin_path=$WORKSPACE/eloqsql_src/bld/data_substrate/host_manager|g" $WORKSPACE/eloqsql_src/concourse/scripts/mtr_multi_bootstrap_ds.cnf
echo "mtr_multi_bootstrap_ds.cnf"
cat $WORKSPACE/eloqsql_pr/concourse/scripts/mtr_multi_bootstrap_ds.cnf
cat $WORKSPACE/eloqsql_src/concourse/scripts/mtr_multi_bootstrap_ds.cnf
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Config edits now target eloqsql_src while tests still use eloqsql_pr.

These edits update configs under eloqsql_src, but later commands (e.g., the MTR bootstrap and dss_server config usage) still reference eloqsql_pr, so the configs actually used by tests may remain unmodified. Please align all config edits and runtime references to the same root.

If the PR flow should continue using eloqsql_pr, consider switching these paths back:

🔧 Possible fix (align to eloqsql_pr)
-update_config_template "$WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap_ds.cnf"
-sed -i "s|eloq_config=.*|eloq_config=$WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap_ds.cnf|g" $WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap.cnf
-sed -i "s|hm_bin_path=.*|hm_bin_path=$WORKSPACE/eloqsql_src/bld/data_substrate/host_manager|g" $WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap_ds.cnf
+update_config_template "$WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap_ds.cnf"
+sed -i "s|eloq_config=.*|eloq_config=$WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap_ds.cnf|g" $WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap.cnf
+sed -i "s|hm_bin_path=.*|hm_bin_path=$WORKSPACE/eloqsql_pr/bld/data_substrate/host_manager|g" $WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap_ds.cnf
🤖 Prompt for AI Agents
In `@concourse/scripts/pr.ent.bash` around lines 105 - 117, The config edits in
pr.ent.bash update templates and sed replacements under eloqsql_src
(update_config_template for mtr_bootstrap_ds.cnf and mtr_multi_bootstrap_ds.cnf
and the sed lines that set eloq_config and hm_bin_path), but later runtime/test
steps still reference eloqsql_pr; to fix, choose one root and make all
references consistent—either change these occurrences of
"$WORKSPACE/eloqsql_src/..." to "$WORKSPACE/eloqsql_pr/..." (for
mtr_bootstrap_ds.cnf, mtr_multi_bootstrap_ds.cnf and hm_bin_path) or update the
downstream MTR/bootstrap and dss_server invocation to point at eloqsql_src;
ensure the same variable/root is used for update_config_template, the sed
replacements (eloq_config and hm_bin_path), and any later test/runtime
references so the configs actually used by tests are the ones you edited.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@concourse/scripts/pr.ent.bash`:
- Around line 169-174: The second cmake invocation overwrites the first, so
ensure only the intended ELOQDSS_ELOQSTORE configuration is used for building
dss_server: remove the second cmake line that passes
-DWITH_DATA_STORE=ELOQDSS_ROCKSDB_CLOUD_S3 or alternatively create a separate
build directory (e.g., bld_rocks or bld_elq) and run the respective cmake
command there; locate the cmake calls in the dss_server build block (mkdir bld
&& cd bld and the two cmake -DWITH_DATA_STORE=... lines) and keep only the cmake
call that sets -DWITH_DATA_STORE=ELOQDSS_ELOQSTORE or split into distinct build
directories before cmake to avoid configuration override.
♻️ Duplicate comments (2)
concourse/scripts/pr.ent.bash (2)

105-117: Align edited bootstrap configs with the paths used at runtime.
These edits target eloqsql_src, but later MTR/bootstrap usage still points at eloqsql_pr, so the tests may consume unmodified configs.

🔧 Possible fix (align to eloqsql_pr)
-update_config_template "$WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap_ds.cnf"
-sed -i "s|eloq_config=.*|eloq_config=$WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap_ds.cnf|g" $WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap.cnf
-sed -i "s|hm_bin_path=.*|hm_bin_path=$WORKSPACE/eloqsql_src/bld/data_substrate/host_manager|g" $WORKSPACE/eloqsql_src/concourse/scripts/mtr_bootstrap_ds.cnf
+update_config_template "$WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap_ds.cnf"
+sed -i "s|eloq_config=.*|eloq_config=$WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap_ds.cnf|g" $WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap.cnf
+sed -i "s|hm_bin_path=.*|hm_bin_path=$WORKSPACE/eloqsql_pr/bld/data_substrate/host_manager|g" $WORKSPACE/eloqsql_pr/concourse/scripts/mtr_bootstrap_ds.cnf

121-127: Remove whitespace in eloq_store_cloud_access_key assignment.
The extra space after = can trip parsers that don't trim whitespace and is inconsistent with the secret key line.

🔧 Proposed fix
-sed -i "s/eloq_store_cloud_access_key.*=.\+/eloq_store_cloud_access_key =${ELOQ_AWS_ACCESS_KEY_ID}/g" $WORKSPACE/eloqsql_src/concourse/scripts/dss_server.ini
+sed -i "s/eloq_store_cloud_access_key.*=.\+/eloq_store_cloud_access_key=${ELOQ_AWS_ACCESS_KEY_ID}/g" $WORKSPACE/eloqsql_src/concourse/scripts/dss_server.ini
📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2b09b9f and ba6e027.

📒 Files selected for processing (3)
  • concourse/scripts/mtr_bootstrap.cnf
  • concourse/scripts/mtr_multi_bootstrap.cnf
  • concourse/scripts/pr.ent.bash
✅ Files skipped from review due to trivial changes (1)
  • concourse/scripts/mtr_multi_bootstrap.cnf
🚧 Files skipped from review as they are similar to previous changes (1)
  • concourse/scripts/mtr_bootstrap.cnf
🧰 Additional context used
🧠 Learnings (4)
📓 Common learnings
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 154
File: storage/eloq/ha_eloq.cc:2571-2623
Timestamp: 2025-11-07T07:10:40.633Z
Learning: Repo: eloqdata/eloqsql PR: 154
File/Area: storage/eloq/ha_eloq.cc (eloq_init_func cluster membership resolution)
Learning: In this repo’s ng_config, the local node is guaranteed to be a candidate (NodeConfig::is_candidate_ == true). Therefore, assigning native_ng_id only when is_candidate_ is true is acceptable; the key improvement is to break early from both loops once the local node is found.
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 127
File: storage/eloq/build_log_service.cmake:97-124
Timestamp: 2025-09-25T11:41:40.111Z
Learning: The `storage/eloq/build_log_service.cmake` file is specifically for the open log service build and only supports ROCKSDB as its log state backend. Unlike the enterprise version in `build_eloq_log_service.cmake`, it does not need to handle cloud-specific variants like ROCKSDB_CLOUD_S3 or ROCKSDB_CLOUD_GCS.
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 154
File: storage/eloq/ha_eloq.cc:2770-2781
Timestamp: 2025-11-07T05:12:43.208Z
Learning: In eloqdata/eloqsql, storage/eloq/ha_eloq.cc (KV_ELOQDS path): for DATA_STORE_TYPE_ELOQDSS_ROCKSDB (local RocksDB), each eloqsql instance owns its own RocksDB instance, and passing create_if_missing=true to DataStoreService::StartService is intentional and safe.
📚 Learning: 2025-09-25T11:41:40.111Z
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 127
File: storage/eloq/build_log_service.cmake:97-124
Timestamp: 2025-09-25T11:41:40.111Z
Learning: The `storage/eloq/build_log_service.cmake` file is specifically for the open log service build and only supports ROCKSDB as its log state backend. Unlike the enterprise version in `build_eloq_log_service.cmake`, it does not need to handle cloud-specific variants like ROCKSDB_CLOUD_S3 or ROCKSDB_CLOUD_GCS.

Applied to files:

  • concourse/scripts/pr.ent.bash
📚 Learning: 2025-11-07T05:12:43.208Z
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 154
File: storage/eloq/ha_eloq.cc:2770-2781
Timestamp: 2025-11-07T05:12:43.208Z
Learning: In eloqdata/eloqsql, storage/eloq/ha_eloq.cc (KV_ELOQDS path): for DATA_STORE_TYPE_ELOQDSS_ROCKSDB (local RocksDB), each eloqsql instance owns its own RocksDB instance, and passing create_if_missing=true to DataStoreService::StartService is intentional and safe.

Applied to files:

  • concourse/scripts/pr.ent.bash
📚 Learning: 2025-09-25T11:33:33.221Z
Learnt from: githubzilla
Repo: eloqdata/eloqsql PR: 127
File: concourse/scripts/build_tarball_open.bash:136-136
Timestamp: 2025-09-25T11:33:33.221Z
Learning: The open log service (OPEN_LOG_SERVICE=ON) in concourse/scripts/build_tarball_open.bash only supports ROCKSDB as its log state and does not use the WITH_LOG_STATE parameter that was introduced to replace USE_ROCKSDB_LOG_STATE in other parts of the codebase.

Applied to files:

  • concourse/scripts/pr.ent.bash
🧬 Code graph analysis (1)
concourse/scripts/pr.ent.bash (2)
concourse/scripts/main.ent.bash (1)
  • update_config_template (49-69)
concourse/scripts/main.bash (1)
  • update_config_template (44-64)
🔇 Additional comments (2)
concourse/scripts/pr.ent.bash (2)

69-89: Config template updates look consistent with eloq_store/MinIO migration.
The added substitutions for eloq_store keys and txlog URL composition are aligned with the new config model.


139-155: ELOQDSS_ELOQSTORE CMake flag looks right.
The top-level CMake configuration now aligns with the new backend selection.

✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.

@githubzilla
Copy link
Collaborator Author

@githubzilla githubzilla merged commit 5349c86 into eloqdata:eloq-10.6.10 Jan 16, 2026
1 of 2 checks passed
@coderabbitai coderabbitai bot mentioned this pull request Jan 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants