Skip to content

chore: update data_substrate#159

Merged
thweetkomputer merged 1 commit intoeloqdata:eloq-10.6.10from
thweetkomputer:update-submodule-zc
Feb 9, 2026
Merged

chore: update data_substrate#159
thweetkomputer merged 1 commit intoeloqdata:eloq-10.6.10from
thweetkomputer:update-submodule-zc

Conversation

@thweetkomputer
Copy link
Collaborator

@thweetkomputer thweetkomputer commented Nov 10, 2025

Summary by CodeRabbit

  • Chores
    • Subproject metadata reference updated; no user-visible behavior change.
  • Configuration
    • Increased store open-files limit and pages-per-file setting; removed obsolete concurrent-writes setting across environments and installer configs.
  • Tests
    • COUNT(*) queries in tests now retry on transient DB conflicts to reduce flakiness and improve stability.
  • Impact
    • No changes to public APIs; production runtime behavior for users remains unchanged.

@thweetkomputer thweetkomputer changed the title update submodule update submodule: refactor store_handler helper functions such as parse_size Nov 10, 2025
@coderabbitai
Copy link

coderabbitai bot commented Nov 10, 2025

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.
  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Bumped data_substrate submodule pointer; increased eloq_store_open_files_limit (256 → 800) and eloq_store_pages_per_file_shift (1 → 2) across Concourse and storage CNF files; removed eloq_store_max_concurrent_writes=0; test scripts now retry SELECT COUNT(*) up to 5 times on MySQL errno 212 with 0.5s backoff and temporary log suppression.

Changes

Cohort / File(s) Summary
Submodule reference
data_substrate
Updated submodule commit pointer (metadata-only change).
Concourse scripts
concourse/scripts/dss_server.ini, concourse/scripts/mtr_bootstrap_ds.cnf, concourse/scripts/mtr_multi_bootstrap_ds.cnf
Increased eloq_store_open_files_limit (256 → 800), changed eloq_store_pages_per_file_shift (1 → 2), and removed eloq_store_max_concurrent_writes=0 where present.
Storage CNF (mono_basic / mono_main)
storage/eloq/mysql-test/mono_basic/data_substrate.cnf, storage/eloq/mysql-test/mono_main/data_substrate.cnf
Updated eloq_store_open_files_limit (256 → 800) and eloq_store_pages_per_file_shift (1 → 2); removed eloq_store_max_concurrent_writes.
Storage CNF (mono_multi variants)
storage/eloq/mysql-test/mono_multi/data_substrate1.cnf, storage/eloq/mysql-test/mono_multi/data_substrate2.cnf, storage/eloq/mysql-test/mono_multi/data_substrate3.cnf
Removed eloq_store_max_concurrent_writes=0 and updated eloq_store_pages_per_file_shift (1 → 2).
Tests - COUNT(*) retry logic
storage/eloq/mysql-test/mono_basic/t/range_split_keycache.test, storage/eloq/mysql-test/mono_multi/t/range_split_keycache.test
Replaced direct SELECT COUNT(*) with a prepared count statement plus a retry loop handling MySQL errno 212: up to 5 retries, 0.5s sleep, temporary suppression of logging, and final echo of count result.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant Test as Test Script
    participant DB as MySQL / Eloq Store
    participant Log as Logger

    rect rgba(200,200,255,0.5)
    Test->>Log: disable logging
    end

    rect rgba(200,255,200,0.5)
    Test->>DB: execute prepared COUNT(*) (query_get_value)
    alt success (no errno)
        DB-->>Test: return count
    else errno 212 (transient conflict)
        DB-->>Test: error 212
        Test->>Test: decrement retry counter, sleep 0.5s, retry
        loop up to 5 times
            Test->>DB: retry COUNT(*)
        end
        alt exhausted
            Test->>Test: abort with fatal error
        end
    end
    end

    rect rgba(255,200,200,0.5)
    Test->>Log: re-enable logging
    Test->>Test: echo final count and query
    end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested reviewers

  • liunyl

Poem

🐇 I nudged a pointer, widened the gate,
Two pages now dance where one stayed late.
When counts collide and queries stall,
Five little hops — a soft half‑second call.
🌱 — cheers from a tuning rabbit.

🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 inconclusive)
Check name Status Explanation Resolution
Title check ❓ Inconclusive The title 'chore: update data_substrate' is vague and overly generic. While it references data_substrate, it doesn't specify what changes are being made or why. The PR involves configuration tuning across multiple files (eloq_store settings, retry mechanisms) plus a submodule update, but the title conveys no meaningful detail about these changes. Use a more specific title that captures the primary change, such as 'chore: tune eloq_store configuration and add retry mechanism for transient conflicts' or 'chore: update data_substrate submodule and optimize storage settings'.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@thweetkomputer thweetkomputer changed the title update submodule: refactor store_handler helper functions such as parse_size chore: update submodule data_substrate Jan 20, 2026
@thweetkomputer thweetkomputer changed the title chore: update submodule data_substrate chore: update submodule eloqstore: fix droptable Feb 3, 2026
@thweetkomputer thweetkomputer force-pushed the update-submodule-zc branch 3 times, most recently from b40b233 to 7b94dbb Compare February 5, 2026 06:20
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@storage/eloq/mysql-test/mono_basic/t/range_split_keycache.test`:
- Around line 31-52: The retry loop currently checks query_get_value and then
has separate if blocks that cause the final if ($mysql_errno) to run even when
$mysql_errno == 212; change the flow in the block using $count_retry,
query_get_value($count_stmt,...), $mysql_errno so that the unexpected-error
branch excludes errno 212 (e.g. convert the final "if ($mysql_errno)" to "else
if ($mysql_errno && $mysql_errno != 212)" or simply make it an "else if" after
the 212-handling branch) and apply the same fix to the other retry blocks at
ranges 80-102, 130-152, and 178-200 to prevent immediate abort on retryable
errno 212.

In `@storage/eloq/mysql-test/mono_multi/t/range_split_keycache.test`:
- Around line 33-54: The retry loop treats errno 212 as an unexpected error
because the final "if ($mysql_errno)" branch runs even when $mysql_errno == 212;
change the logic so the unexpected-error branch excludes 212 (e.g., use "else if
($mysql_errno != 212)" or make it an "else if" after the 212-handling block) in
the loop around query_get_value/$count_stmt/$count_retry so the dec $count_retry
retry path for errno 212 can execute; apply the same change to the other retry
blocks referenced at the other ranges (lines ~82-104, ~132-154, ~180-202) that
use $mysql_errno, dec $*_retry and the unexpected-error die.

Comment on lines +31 to +52
while($count_retry)
{
let $count_value= query_get_value($count_stmt, count(*), 1);
if (!$mysql_errno)
{
let $count_retry= 0;
}
if ($mysql_errno == 212)
{
dec $count_retry;
if (!$count_retry)
{
--enable_abort_on_error
--die "SELECT COUNT(*) failed with read/write conflict after 5 retries";
}
--sleep 0.5
}
if ($mysql_errno)
{
--enable_abort_on_error
--die "SELECT COUNT(*) failed with unexpected error";
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Retry loop still aborts immediately on errno 212.

The final if ($mysql_errno) triggers even when $mysql_errno == 212, so the script dies before any retry. Make the “unexpected error” branch exclude 212 (or convert to else if).

🔧 Suggested fix (apply to each retry block)
-    if ($mysql_errno)
+    if ($mysql_errno && $mysql_errno != 212)
     {
         --enable_abort_on_error
         --die "SELECT COUNT(*) failed with unexpected error";
     }

Also applies to: 80-102, 130-152, 178-200

🤖 Prompt for AI Agents
In `@storage/eloq/mysql-test/mono_basic/t/range_split_keycache.test` around lines
31 - 52, The retry loop currently checks query_get_value and then has separate
if blocks that cause the final if ($mysql_errno) to run even when $mysql_errno
== 212; change the flow in the block using $count_retry,
query_get_value($count_stmt,...), $mysql_errno so that the unexpected-error
branch excludes errno 212 (e.g. convert the final "if ($mysql_errno)" to "else
if ($mysql_errno && $mysql_errno != 212)" or simply make it an "else if" after
the 212-handling branch) and apply the same fix to the other retry blocks at
ranges 80-102, 130-152, and 178-200 to prevent immediate abort on retryable
errno 212.

Comment on lines +33 to +54
while($count_retry)
{
let $count_value= query_get_value($count_stmt, count(*), 1);
if (!$mysql_errno)
{
let $count_retry= 0;
}
if ($mysql_errno == 212)
{
dec $count_retry;
if (!$count_retry)
{
--enable_abort_on_error
--die "SELECT COUNT(*) failed with read/write conflict after 5 retries";
}
--sleep 0.5
}
if ($mysql_errno)
{
--enable_abort_on_error
--die "SELECT COUNT(*) failed with unexpected error";
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Retry loop still aborts immediately on errno 212.

The generic error branch fires for 212, so the retry never occurs. Gate the “unexpected error” branch to exclude 212 (or use else if).

🔧 Suggested fix (apply to each retry block)
-    if ($mysql_errno)
+    if ($mysql_errno && $mysql_errno != 212)
     {
         --enable_abort_on_error
         --die "SELECT COUNT(*) failed with unexpected error";
     }

Also applies to: 82-104, 132-154, 180-202

🤖 Prompt for AI Agents
In `@storage/eloq/mysql-test/mono_multi/t/range_split_keycache.test` around lines
33 - 54, The retry loop treats errno 212 as an unexpected error because the
final "if ($mysql_errno)" branch runs even when $mysql_errno == 212; change the
logic so the unexpected-error branch excludes 212 (e.g., use "else if
($mysql_errno != 212)" or make it an "else if" after the 212-handling block) in
the loop around query_get_value/$count_stmt/$count_retry so the dec $count_retry
retry path for errno 212 can execute; apply the same change to the other retry
blocks referenced at the other ranges (lines ~82-104, ~132-154, ~180-202) that
use $mysql_errno, dec $*_retry and the unexpected-error die.

@thweetkomputer thweetkomputer changed the title chore: update submodule eloqstore: fix droptable chore: update data_substrate Feb 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants