HDDS-12336. Support Prefix, Delimiter, and CommonPrefixes in ListMultipartUploads by 0lai0 · Pull Request #9286 · apache/ozone

0lai0 · 2025-11-12T12:05:24Z

What changes were proposed in this pull request?

To enhance the AWS behavior of the ListMultipartUploads API in Ozone S3 Gateway:

Add delimiter and encoding-type query parameters, and include Delimiter, EncodingType, and CommonPrefixes in the response.
The response class ListMultipartUploadsResult now wraps Prefix and Delimiter with EncodingTypeObject, and supports CommonPrefixes.
The BucketEndpoint's list logic groups multiple file uploads using the ListObjects/V2 method, correctly generating common prefixes.

What is the link to the Apache JIRA

HDDS-12336

…partInfoInsightHandler

…w signature

ivandika3 · 2025-11-13T05:11:49Z

@echonesis Please also help take a look.

echonesis

@0lai0 Thanks for the patch, left some comments.

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java

.../s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ListMultipartUploadsResult.java

0lai0 · 2025-11-13T15:50:11Z

Thanks @echonesis for pointing this out. I'll get on this.

…& NextKeyMarker

chungen0126

Thanks @0lai0 for the patch. Left some comments

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java

chungen0126

+1 LGTM

peterxcli

Thanks for this patch, left some comments.

Few thoughts that I didnt list in the code comments:

lets add a builder for list multipart uploads response class, and move the complex common prefix extracting logic into it.
Please add some test case coverage into AWS SDK IT

.../s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ListMultipartUploadsResult.java

peterxcli · 2025-11-18T14:04:53Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java

-          new ListMultipartUploadsResult.Upload(
-              upload.getKeyName(),
+      String prevDir = null;
+      for (OzoneMultipartUpload upload : ozoneMultipartUploadList.getUploads()) {


These extract common prefix should only happen if delimiter has value

peterxcli · 2025-11-18T14:06:00Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java

+      String prevDir = null;
+      for (OzoneMultipartUpload upload : ozoneMultipartUploadList.getUploads()) {
+        String keyName = upload.getKeyName();
+        if (bucket.getBucketLayout().isFileSystemOptimized() &&


Please could you add some comment to explain the logic here? thanks very much!

peterxcli · 2025-11-18T14:07:36Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java

+        if (!addedAsPrefix) {
+          result.addUpload(new ListMultipartUploadsResult.Upload(


Is this intentional? some records return from OM would disappear

Yes, When delimiter is provided, uploads sharing a common prefix are collapsed into CommonPrefixes instead of appearing individually in Uploads. So I think it would not disappear.

But I wasn't 100% sure about this part. Does this implementation look correct ? If I still get incorrect way , could you provide some guidance? Thank you so much.

0lai0 · 2025-11-20T07:02:39Z

Thanks @peterxcli . I'll get on this.

ivandika3 · 2025-11-24T01:03:24Z

Thanks @0lai0 for the patch, I'll take a look soon. Meanwhile, please kindly refer to the https://issues.apache.org/jira/browse/HDDS-12882 comment regarding the case where Delimiter is specified (i.e. how to derive CommonPrefixes and Contents).

0lai0 · 2025-11-24T14:31:05Z

Thanks @ivandika3 for pointing this out. I'll look into this.

ivandika3 · 2025-11-24T14:33:08Z

@0lai0 Thanks, please also add similar tests in TestBucketList and also add pagination tests to AWS SDK integration tests (i.e. AbstractS3SDKV1Tests and AbstractS3SDKV2Tests).

0lai0 · 2025-11-24T15:32:22Z

Thank you. I'll do some research on how to approach this.

github-actions · 2025-12-26T00:05:50Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

echonesis

Thanks @0lai0 for the patch.

echonesis · 2026-01-26T03:20:30Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java

+      if (keyName.length() < normalizedPrefix.length()) {
+        continue;
+      }


This check is redundant when getBucketLayout().isFileSystemOptimized() is true
(covered by L450's startsWith check).

Should the prefix filtering apply to all bucket layouts?

echonesis · 2026-01-26T03:31:53Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java

+            && currentDirName.equals(prevDir)) {
+          lastProcessedKey = keyName;
+          lastProcessedUploadId = upload.getUploadId();
+          continue;


processedUploads won't be added in this condition.
Is that what we want?

github-actions · 2026-02-17T00:08:35Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

adoroszlai · 2026-02-20T10:16:44Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java

+      final String aclMarker = queryParams().get(QueryParams.ACL);
+      if (aclMarker != null) {
+        s3GAction = S3GAction.GET_ACL;
+        S3BucketAcl result = getAcl(bucketName);
+        getMetrics().updateGetAclSuccessStats(startNanos);
+        auditReadSuccess(s3GAction);
+        return Response.ok(result, MediaType.APPLICATION_XML_TYPE).build();
+      }


Please don't restore this, it was moved to BucketAclHandler in HDDS-14360.

adoroszlai · 2026-02-20T10:20:45Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java

+      String delimiter,
+      String encodingType,


listMultipartUploads call does not pass these new arguments, causing compile error.

adoroszlai · 2026-02-20T10:21:39Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java

+      AUDIT.logReadSuccess(buildAuditMessageForSuccess(s3GAction,
+          getAuditParameters()));


Please keep simplified auditReadSuccess(s3GAction) call.

github-actions · 2026-03-23T00:09:20Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

0lai0 added 3 commits November 12, 2025 12:02

HDDS-13490. Implement KeyInfoProtoLight-based PartKeyInfoMap in Multi…

a342fe2

…partInfoInsightHandler

fix checkstyle

8a31862

Fix TestPermissionCheck: update listMultipartUploads call to match ne…

4d220d3

…w signature

peterxcli self-requested a review November 12, 2025 12:28

peterxcli added the s3 S3 Gateway label Nov 12, 2025

echonesis reviewed Nov 13, 2025

View reviewed changes

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java Outdated Show resolved Hide resolved

.../s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ListMultipartUploadsResult.java Show resolved Hide resolved

Merge branch 'apache:master' into HDDS-12336

4b4f62c

0lai0 and others added 2 commits November 14, 2025 12:39

Merge branch 'apache:master' into HDDS-12336

6d52398

fix consolidate duplicate conditional fragments and modify KeyMarker …

c7f70f8

…& NextKeyMarker

chungen0126 reviewed Nov 15, 2025

View reviewed changes

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java Outdated Show resolved Hide resolved

0lai0 and others added 4 commits November 15, 2025 21:40

Merge branch 'apache:master' into HDDS-12336

69adaa2

move repeat function

55e3aa8

Merge branch 'apache:master' into HDDS-12336

4bac14a

modify key in upload to encodinf type

e019d5e

chungen0126 approved these changes Nov 18, 2025

View reviewed changes

peterxcli requested changes Nov 18, 2025

View reviewed changes

peterxcli requested a review from ivandika3 November 18, 2025 14:12

Merge branch 'apache:master' into HDDS-12336

74ee47e

0lai0 and others added 4 commits November 21, 2025 16:56

Merge branch 'apache:master' into HDDS-12336

2032769

WIP-chang function name make it clear

72e13d7

Merge branch 'apache:master' into HDDS-12336

bc0931d

add modify

6ee3253

Merge branch 'apache:master' into HDDS-12336

43eddb4

0lai0 added 2 commits November 27, 2025 17:28

add test in bucketlist and pagination

d25c61c

fix checkstyle

be4f355

0lai0 requested a review from peterxcli December 4, 2025 09:07

github-actions bot added the stale label Dec 26, 2025

Merge branch 'master' into HDDS-12336

0bbfcdc

github-actions bot removed the stale label Dec 27, 2025

Merge branch 'master' into HDDS-12336

0c6da36

0lai0 requested a review from echonesis January 19, 2026 15:09

echonesis reviewed Jan 26, 2026

View reviewed changes

github-actions bot added the stale label Feb 17, 2026

adoroszlai reviewed Feb 20, 2026

View reviewed changes

github-actions bot removed the stale label Feb 21, 2026

0lai0 marked this pull request as draft March 1, 2026 02:40

github-actions bot added the stale label Mar 23, 2026

Merge branch 'apache:master' into HDDS-12336

90d6032

github-actions bot removed the stale label Mar 25, 2026

		if (!addedAsPrefix) {
		result.addUpload(new ListMultipartUploadsResult.Upload(

		AUDIT.logReadSuccess(buildAuditMessageForSuccess(s3GAction,
		getAuditParameters()));

Conversation

0lai0 commented Nov 12, 2025

What changes were proposed in this pull request?

What is the link to the Apache JIRA

Uh oh!

ivandika3 commented Nov 13, 2025

Uh oh!

echonesis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

0lai0 commented Nov 13, 2025

Uh oh!

chungen0126 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chungen0126 left a comment

Choose a reason for hiding this comment

Uh oh!

peterxcli left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0lai0 commented Nov 20, 2025

Uh oh!

ivandika3 commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0lai0 commented Nov 24, 2025

Uh oh!

ivandika3 commented Nov 24, 2025

Uh oh!

0lai0 commented Nov 24, 2025

Uh oh!

github-actions bot commented Dec 26, 2025

Uh oh!

echonesis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

peterxcli left a comment •

edited

Loading

ivandika3 commented Nov 24, 2025 •

edited

Loading