HDDS-13108. Refactor StorageVolume to use SlidingWindow by ptlrs · Pull Request #8843 · apache/ozone

ptlrs · 2025-07-22T06:37:23Z

Please describe your PR in detail:

This PR uses the new sliding window implementation.
It migrates all existing checks to detect a failed volume to use the new time-based sliding window utility.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13108

How was this patch tested?

CI:https://github.com/ptlrs/ozone/actions/runs/16436635030

… scanner failures

… tests

…e-failed-volume-checks-to-one-sliding-window

…g window mechanism

ptlrs · 2025-07-22T18:03:33Z

Hi @errose28 @Tejaskriya @adoroszlai can you please review this PR?

Tejaskriya

Thanks for working on this @ptlrs , please find a suggestion below

...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java

ptlrs · 2025-07-31T06:38:04Z

Thanks for the review @Tejaskriya. I have added the configuration.

Tejaskriya

Looks good to me, @errose28 could you please take a look?

github-actions · 2025-11-11T00:05:52Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

github-actions · 2025-11-25T00:05:10Z

Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.

ptlrs · 2026-02-09T19:29:33Z

Hi @errose28, could you please reopen this PR?

…e-failed-volume-checks-to-one-sliding-window # Conflicts: # hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java # hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java

ptlrs · 2026-02-12T08:48:18Z

Hi @errose28, the conflicts have been resolved for this PR. Could you please take a look.

github-actions · 2026-03-06T00:13:27Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

…failed-volume-checks-to-one-sliding-window

adoroszlai

Thanks @ptlrs for the patch.

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java

...test/java/org/apache/hadoop/ozone/container/common/volume/TestStorageVolumeHealthChecks.java

ChenSammi · 2026-03-11T04:34:12Z

...c/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java

+          + " failure result stored in the sliding window will expire."
+          + " Unit could be defined with postfix (ns,ms,s,m,h,d)."
+  )
+  private Duration diskCheckSlidingWindowTimeout = DISK_CHECK_SLIDING_WINDOW_TIMEOUT_DEFAULT;


What's the recommendation of value relationship between this new property and PERIODIC_DISK_CHECK_INTERVAL_MINUTES_DEFAULT? Say if user reconfigured PERIODIC_DISK_CHECK_INTERVAL_MINUTES_DEFAULT to 2h, or 30m, shall we suggest user to reconfigure this property too?

This is a good question.

The period disk check currently runs every 1 hour and the sliding window coverage is also for 1 hour.

The window should likely cover the period between and inclusive of the two periodic checks such that if two periodic checks fail, then they get counted in the same window.

Otherwise the only opportunity for a failure will be by a combination of periodic and on-demand checks and never due to two periodic checks.

I have updated the sliding window to be as long as the periodic disk check interval plus the timeout value for the disk check.

...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java

ChenSammi · 2026-03-11T04:43:26Z

...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java

    }

+    LOG.debug("IO test results for volume {}: encountered {} out of {} tolerated failures",
+        this, ioTestSlidingWindow.getNumEventsInWindow(), ioTestSlidingWindow.getWindowSize());


Shall we call getNumEvents() here, instead of getNumEventsInWindow()?

getNumEventsInWindow is correct here. It will show how many failures are in the actual window. The window size is the number of failures that are allowed.

This log line is logged every time in debug mode. Since our queue is larger than the window, logging getNumEvents may give us the false impression that we got more failures as it will include the errors outside the window as well.

We are using getNumEvents only for logging in tests.

…k interval

…stomizable sliding window timestamps

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/SlidingWindow.java

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

ptlrs added 4 commits July 11, 2025 22:15

HDDS-13108. Transition to use the generic sliding window for counting…

b1df449

… scanner failures

HDDS-13108. Remove unused SlidingWindow implementation and associated…

625d52a

… tests

Merge remote-tracking branch 'upstream/master' into HDDS-13108-Migrat…

f108d3c

…e-failed-volume-checks-to-one-sliding-window

HDDS-13108. Replace count-based failure checks with time-based slidin…

33b9a4b

…g window mechanism

ptlrs marked this pull request as draft July 22, 2025 06:38

errose28 added the scanners Changes related to datanode container and volume scanners label Jul 22, 2025

Tejaskriya self-requested a review July 23, 2025 08:31

errose28 self-requested a review July 23, 2025 18:46

Tejaskriya reviewed Jul 29, 2025

View reviewed changes

...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java Outdated Show resolved Hide resolved

HDDS-13108. Add configuration for sliding window timeout in disk checks

e81f59b

ptlrs requested a review from Tejaskriya July 31, 2025 06:38

Tejaskriya reviewed Aug 4, 2025

View reviewed changes

github-actions bot added the stale label Nov 11, 2025

github-actions bot closed this Nov 25, 2025

errose28 reopened this Feb 9, 2026

github-actions bot removed the stale label Feb 10, 2026

HDDS-13108. Remove unused imports from HddsVolume

17fa938

github-actions bot added the stale label Mar 6, 2026

Merge remote-tracking branch 'origin/master' into HDDS-13108-Migrate-…

4d8b6fe

…failed-volume-checks-to-one-sliding-window

adoroszlai reviewed Mar 9, 2026

View reviewed changes

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java Outdated Show resolved Hide resolved

...test/java/org/apache/hadoop/ozone/container/common/volume/TestStorageVolumeHealthChecks.java Outdated Show resolved Hide resolved

adoroszlai removed the stale label Mar 9, 2026

adoroszlai changed the title ~~HDDS-13108. Migrate failed volume checks to one sliding window~~ HDDS-13108. Refactor StorageVolume to use SlidingWindow Mar 9, 2026

checkstyle

0b9e02b

ChenSammi reviewed Mar 11, 2026

View reviewed changes

...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java Show resolved Hide resolved

ChenSammi reviewed Mar 11, 2026

View reviewed changes

ptlrs added 3 commits March 24, 2026 19:39

HDDS-13108. Enforce minimum sliding window timeout based on disk chec…

158819e

…k interval

HDDS-13108. Add Clock support to StorageVolume and its Builder for cu…

10d0dcd

…stomizable sliding window timestamps

HDDS-13108. Update sliding window checks and fix test clock adjustments

af0e4f8

adoroszlai reviewed Mar 25, 2026

View reviewed changes

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/SlidingWindow.java Outdated Show resolved Hide resolved

Update SlidingWindow.java

1ec9e0e

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

Conversation

ptlrs commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Please describe your PR in detail:

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

ptlrs commented Jul 22, 2025

Uh oh!

Tejaskriya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ptlrs commented Jul 31, 2025

Uh oh!

Tejaskriya left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 11, 2025

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

ptlrs commented Feb 9, 2026

Uh oh!

ptlrs commented Feb 12, 2026

Uh oh!

github-actions bot commented Mar 6, 2026

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ChenSammi Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

ptlrs Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ChenSammi Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

ptlrs Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ptlrs commented Jul 22, 2025 •

edited

Loading