fix: GI-18072 - Allow retries on marker file creation requests #18073

PavithranRick · 2026-02-02T20:36:45Z

Describe the issue this Pull Request addresses

rollback gets triggered (internally) when clean() is called. Rollbacks are failed due to marker file creation retries. #18072

When using remote marker files, if a request times out but the marker is actually created successfully, the subsequent retry fails because the marker already exists. Retries in this scenario should succeed.

Summary and Changelog

This PR improves retry behavior for remote marker file creation by making marker operations idempotent.

Handles timeout scenarios where the marker was created but the client did not receive an acknowledgement.
Introduces a request ID (or similar tracking mechanism) to safely support idempotent retries without creating additional risks.

Impact

Reduces rollback failures (via cleaner) caused by non-idempotent marker retries and improves robustness of remote marker handling under transient network issues.

Risk Level

low — change is scoped to retry/idempotency behavior for remote marker requests. Verification includes ensuring retries succeed when markers already exist due to previous timed-out requests.

Documentation Update

none

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

the-other-tim-brown · 2026-02-03T03:20:33Z

...e-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java

+        // Idempotent retry: marker already created with same requestId
+        if (markerToRequestIdMap.containsKey(markerName)) {
+          String existingRequestId = markerToRequestIdMap.get(markerName);
+          String normalizedRequestId = requestId == null ? NULL_REQUEST_ID : requestId;


If the requestId is null we should just skip this check.

how can incoming requestId could be null.
when this release goes out, any marker request to TLS should have a request Id set right?

the-other-tim-brown · 2026-02-03T03:22:05Z

...e-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java

+          continue;
        }
-        future.setIsSuccessful(!exists);
+        if (conflictDetectionStrategy.isPresent()) {


Just simplify this to else if?

...-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/MarkerHandler.java

the-other-tim-brown · 2026-02-03T03:25:12Z

...e-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java


  /**
   * Syncs all markers maintained in the underlying files under the marker directory in the file system.
+   * When TLS recovers from crash, we have no request IDs so store sentinel value for each marker.


Since we cannot know what the requestID was, we can avoid backfilling this map with the NULL_REQUEST_ID and let the requests fail to be safe.

but to fail the request, we end up looking up in markerToRequestIdMap right. So, to even fail a marker request call to an already existing marker, we need to populate entries in the map.

Right now we will allow the request to succeed if they are both NULL_REQUEST_ID which is different from existing behavior where we fail in this case. I think it is easier to reason about if we don't backfill this map.

nsivabalan · 2026-02-03T17:53:07Z

one minor correction in PR desc. we are fixing the markers retries for rollbacks and not cleaner.
we do not have any markers for cleaner only.
just that, rollback gets triggered (internally) when clean() is called.

nsivabalan · 2026-02-03T17:58:19Z

...e-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java

+        // Idempotent retry: marker already created with same requestId
+        if (markerToRequestIdMap.containsKey(markerName)) {
+          String existingRequestId = markerToRequestIdMap.get(markerName);
+          String normalizedRequestId = requestId == null ? NULL_REQUEST_ID : requestId;


how can incoming requestId could be null.
when this release goes out, any marker request to TLS should have a request Id set right?

...e-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java

nsivabalan · 2026-02-03T18:03:27Z

...e-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java


  /**
   * Syncs all markers maintained in the underlying files under the marker directory in the file system.
+   * When TLS recovers from crash, we have no request IDs so store sentinel value for each marker.


but to fail the request, we end up looking up in markerToRequestIdMap right. So, to even fail a marker request call to an already existing marker, we need to populate entries in the map.

...e-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java

hudi-timeline-service/src/test/java/org/apache/hudi/timeline/service/TestRequestHandler.java

...e-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java

the-other-tim-brown · 2026-02-05T18:47:16Z

...e-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java


  /**
   * Syncs all markers maintained in the underlying files under the marker directory in the file system.
+   * When TLS recovers from crash, we have no request IDs so store sentinel value for each marker.


Right now we will allow the request to succeed if they are both NULL_REQUEST_ID which is different from existing behavior where we fail in this case. I think it is easier to reason about if we don't backfill this map.

hudi-bot · 2026-02-10T02:21:26Z

CI report:

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

GI-18072 - Allow retries on marker files

f6db969

PavithranRick changed the title ~~GI-18072 - Allow retries on marker files~~ fix: GI-18072 - Allow retries on marker files Feb 2, 2026

GI-18072 - Allow retries on marker files - added more tests

be90d4f

PavithranRick changed the title ~~fix: GI-18072 - Allow retries on marker files~~ fix: GI-18072 - Allow retries on marker file creation requests Feb 2, 2026

github-actions bot added the size:M PR with lines of changes in (100, 300] label Feb 2, 2026

the-other-tim-brown reviewed Feb 3, 2026

View reviewed changes

nsivabalan reviewed Feb 3, 2026

View reviewed changes

Pavithran Ravichandiran added 2 commits February 3, 2026 14:36

GI-18072 - Addressed comments

3aa87e0

GI-18072 - Addressed comments

ae5f6a8

the-other-tim-brown reviewed Feb 5, 2026

View reviewed changes

PavithranRick added 3 commits February 9, 2026 14:36

GI-18072 - Addressed comments

cdee329

GI-18072 - Addressed comments

140c3bb

GI-18072 - Addressed comments - around requestId

ddf29f9

fix: GI-18072 - Allow retries on marker file creation requests #18073

Are you sure you want to change the base?

fix: GI-18072 - Allow retries on marker file creation requests #18073

Conversation

PavithranRick commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan commented Feb 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Feb 10, 2026

CI report:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

PavithranRick commented Feb 2, 2026 •

edited

Loading