Skip to content

warm up standby local ssd#288

Merged
liunyl merged 1 commit intoeloqdata:mainfrom
liunyl:standby_warm
Nov 20, 2025
Merged

warm up standby local ssd#288
liunyl merged 1 commit intoeloqdata:mainfrom
liunyl:standby_warm

Conversation

@liunyl
Copy link
Contributor

@liunyl liunyl commented Nov 19, 2025

Note

Adds a configurable dss_file_cache_sync_interval_sec flag/config for DSS standby warm-up and includes s3_file_downloader.cpp in ELOQDSS cloud builds.

  • Config/Runtime (DSS):
    • Introduce dss_file_cache_sync_interval_sec (flag and INI) and pass to DataStoreServiceClusterManager via SetFileCacheSyncIntervalSec(...) for standby warm-up.
  • Build:
    • Include store_handler/eloq_data_store_service/s3_file_downloader.cpp for ELOQDSS_ROCKSDB_CLOUD_* targets.

Written by Cursor Bugbot for commit e1d364c. This will update automatically on new commits. Configure here.

Summary by CodeRabbit

  • New Features

    • Added a configurable file cache synchronization interval, adjustable via command-line flag or configuration and applied during data store initialization across deployment modes.
  • Chores

    • Build updated to include S3 file download support when the cloud S3 data store is selected.
    • Submodule pointer updated with no behavior changes.

✏️ Tip: You can customize this high-level summary in your review settings.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Nov 19, 2025

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

Adds an S3 file downloader to the CMake resources for ELOQDSS_ROCKSDB_CLOUD_S3, introduces a new CLI flag dss_file_cache_sync_interval_sec and applies it to EloqDS configuration in two initialization paths, and updates the store_handler submodule pointer.

Changes

Cohort / File(s) Change Summary
Build configuration
CMakeLists.txt
Adds store_handler/eloq_data_store_service/s3_file_downloader.cpp to RESELOQ_RESOURCES under WITH_DATA_STORE STREQUAL "ELOQDSS_ROCKSDB_CLOUD_S3".
Redis service / DSS config
src/redis_service.cpp
Adds DEFINE_uint32(dss_file_cache_sync_interval_sec, 30, "File cache sync interval in seconds for standby warm-up");. Reads CLI flag when set (otherwise falls back to store.dss_file_cache_sync_interval_sec) and calls ds_config.SetFileCacheSyncIntervalSec(...) in both single-node bootstrap and multi-node EloqDS DSS cluster construction paths.
Submodule update
store_handler
Updates submodule pointer from commit 1c00129c7cc8e83fb9b2237f4289fd76dd2c2fd5 to 87ee52bf0672fee7c8f4b190c93b99cdfcb1bb37.

Sequence Diagram(s)

sequenceDiagram
    participant CLI as CLI / Flags
    participant RedisSvc as RedisService::Init
    participant DSConfig as ds_config
    participant EloqDS as EloqDS Startup

    CLI->>RedisSvc: optional dss_file_cache_sync_interval_sec
    RedisSvc->>DSConfig: construct/populate ds_config
    alt CLI flag set
        CLI->>RedisSvc: use CLI-provided value
    else
        RedisSvc->>DSConfig: read store.dss_file_cache_sync_interval_sec
    end
    RedisSvc->>DSConfig: SetFileCacheSyncIntervalSec(value)
    RedisSvc->>EloqDS: pass ds_config to start data store (single-node or cluster)
    EloqDS->>DSConfig: use file cache sync interval during standby warm-up
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

  • Pay attention to flag precedence logic (explicit CLI vs store config).
  • Review both insertion sites where SetFileCacheSyncIntervalSec is called.
  • Verify the CMake path correctness and conditional gating.
  • Confirm submodule update doesn't require API or build-script adjustments.

Possibly related PRs

Suggested reviewers

  • thweetkomputer
  • xiexiaoy

Poem

🐰 I hop through CMake and flags so bright,
I set the sync seconds, ready for night.
S3 crumbs I fetch, caches warm and near,
Submodule bumped — a twirl and a cheer.
Happy builds and carrots clear! 🥕

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'warm up standby local ssd' directly relates to the PR's main objective of warming up standby instances with local SSD file cache synchronization, which is the core feature being added.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ff4fc4e and e1d364c.

📒 Files selected for processing (3)
  • CMakeLists.txt (1 hunks)
  • src/redis_service.cpp (3 hunks)
  • store_handler (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • store_handler
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/redis_service.cpp
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-09-24T09:00:44.250Z
Learnt from: githubzilla
Repo: eloqdata/eloqkv PR: 193
File: build_log_service.cmake:103-118
Timestamp: 2025-09-24T09:00:44.250Z
Learning: The build_log_service.cmake file is specifically for building the log_service submodule which only supports ROCKSDB log state, unlike build_eloq_log_service.cmake which handles multiple WITH_LOG_STATE options (MEMORY, ROCKSDB, ROCKSDB_CLOUD_S3, ROCKSDB_CLOUD_GCS). Therefore, requiring RocksDB unconditionally in build_log_service.cmake is the correct behavior.

Applied to files:

  • CMakeLists.txt
📚 Learning: 2025-09-26T05:07:45.185Z
Learnt from: githubzilla
Repo: eloqdata/eloqkv PR: 193
File: concourse/scripts/build_tarball_open.bash:121-121
Timestamp: 2025-09-26T05:07:45.185Z
Learning: The open log service in build_tarball_open.bash only supports ROCKSDB as the single log state option, unlike the main eloq log service which supports multiple WITH_LOG_STATE options. Therefore, no WITH_LOG_STATE configuration flag is needed for the open log service build.

Applied to files:

  • CMakeLists.txt
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Cursor Bugbot
🔇 Additional comments (1)
CMakeLists.txt (1)

566-584: Verify scope of s3_file_downloader.cpp: confirm whether it is S3-only or needed for both cloud backends.

The file store_handler/eloq_data_store_service/s3_file_downloader.cpp (line 583) has an S3-specific name, yet it's included in a combined conditional block (lines 566–567) for both ELOQDSS_ROCKSDB_CLOUD_S3 and ELOQDSS_ROCKSDB_CLOUD_GCS. This conflicts with the established build pattern in the codebase, where S3-specific setup uses nested conditionals that separate S3 from GCS (see lines 252–254).

If s3_file_downloader.cpp is truly S3-only, it should be placed in a separate S3-specific conditional block. If it is shared functionality, consider renaming it to reflect its generic purpose and document its role for both backends.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the final PR Bugbot will review for you during this billing cycle

Your free Bugbot reviews will reset on December 19

Details

You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

: config_reader.GetInteger(
"store",
"dss_file_cache_sync_interval_sec",
FLAGS_dss_file_cache_sync_interval_sec);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Negative config value silently converted to large positive

The config_reader.GetInteger() method returns a long (signed type), which is assigned to dss_file_cache_sync_interval_sec of type uint32_t (unsigned). If a user provides a negative value in the configuration file, it will be implicitly converted to a large positive uint32_t value through two's complement representation, potentially causing the file cache sync to occur at an unexpectedly long interval instead of being rejected as invalid.

Fix in Cursor Fix in Web

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
src/redis_service.cpp (1)

1037-1050: Config wiring for dss_file_cache_sync_interval_sec matches existing patterns

The interval is:

  • Taken from the CLI when explicitly set, otherwise from store.dss_file_cache_sync_interval_sec with the flag default as fallback.
  • Applied to EloqDS::DataStoreServiceClusterManager via SetFileCacheSyncIntervalSec(...) after the DSS topology is loaded from either peer or tx configs.

This mirrors other config/flag precedence in Init and ensures every DSS deployment gets a well-defined interval. If you want better debuggability, you could optionally log the effective interval once after it’s computed.

Also applies to: 1100-1102

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between dc1c102 and ff4fc4e.

📒 Files selected for processing (3)
  • CMakeLists.txt (1 hunks)
  • src/redis_service.cpp (3 hunks)
  • store_handler (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • store_handler
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-09-24T09:00:44.250Z
Learnt from: githubzilla
Repo: eloqdata/eloqkv PR: 193
File: build_log_service.cmake:103-118
Timestamp: 2025-09-24T09:00:44.250Z
Learning: The build_log_service.cmake file is specifically for building the log_service submodule which only supports ROCKSDB log state, unlike build_eloq_log_service.cmake which handles multiple WITH_LOG_STATE options (MEMORY, ROCKSDB, ROCKSDB_CLOUD_S3, ROCKSDB_CLOUD_GCS). Therefore, requiring RocksDB unconditionally in build_log_service.cmake is the correct behavior.

Applied to files:

  • CMakeLists.txt
📚 Learning: 2025-09-26T05:07:45.185Z
Learnt from: githubzilla
Repo: eloqdata/eloqkv PR: 193
File: concourse/scripts/build_tarball_open.bash:121-121
Timestamp: 2025-09-26T05:07:45.185Z
Learning: The open log service in build_tarball_open.bash only supports ROCKSDB as the single log state option, unlike the main eloq log service which supports multiple WITH_LOG_STATE options. Therefore, no WITH_LOG_STATE configuration flag is needed for the open log service build.

Applied to files:

  • CMakeLists.txt
🧬 Code graph analysis (1)
src/redis_service.cpp (1)
src/vector/vector_type.cpp (2)
  • CheckCommandLineFlagIsDefault (34-46)
  • CheckCommandLineFlagIsDefault (34-34)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Cursor Bugbot
🔇 Additional comments (2)
src/redis_service.cpp (1)

360-371: New DSS file-cache sync flag is correctly scoped and defaulted

The DEFINE_uint32(dss_file_cache_sync_interval_sec, 30, ...) flag is guarded by #if ELOQDS and follows the existing gflags patterns. Defaulting to 30 seconds and documenting it in the flag description look consistent with the intended standby warm-up behavior.

CMakeLists.txt (1)

565-583: The review comment is based on a false premise: the referenced file does not exist

The repository's store_handler/ directory is empty (0 files), and store_handler/eloq_data_store_service/s3_file_downloader.cpp does not exist anywhere in the codebase. The CMakeLists.txt at lines 565-583 references source files that are not present, which will cause the build to fail when it attempts to compile them.

The original review comment's concern about scoping s3_file_downloader.cpp to S3-only is moot—the file doesn't exist. The actual issue is that the PR modifies CMakeLists.txt to reference non-existent source files, suggesting the PR is incomplete.

Likely an incorrect or invalid review comment.

Copy link
Collaborator

@githubzilla githubzilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@liunyl liunyl merged commit fea82b9 into eloqdata:main Nov 20, 2025
4 of 5 checks passed
@liunyl liunyl deleted the standby_warm branch November 20, 2025 06:39
liunyl added a commit that referenced this pull request Nov 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants