HDDS-13919. S3 Conditional Writes (PutObject) by peterxcli · Pull Request #9815 · apache/ozone

peterxcli · 2026-02-24T11:32:38Z

What changes were proposed in this pull request?

Add createKeyIfNotExists, rewriteKeyIfMatch to RpcClient API, prevent the logic mix wit the existing rewriteKey
Object Endpoint put endpoint handler multiplex createKeyIfNotExists, rewriteKeyIfMatch and createKey
OM side validation logic of etag matching in key creation and commit request
Error translation from OM exception S3 Error for conditional request

ozone/hadoop-hdds/docs/content/design/s3-conditional-requests.md

Lines 59 to 82 in 6c8a08e

    
           ## Specification 
        
           ### AWS S3 Conditional Write Specification 
        
           #### If-None-Match Header 
        
           ``` 
        
           If-None-Match: "*" 
        
           ``` 
        
           - Succeeds only if object does NOT exist 
        
           - Returns `412 Precondition Failed` if object exists 
        
           - Primary use case: Create-only semantics 
        
           #### If-Match Header 
        
           ``` 
        
           If-Match: "<etag>" 
        
           ``` 
        
           - Succeeds only if object EXISTS and ETag matches 
        
           - Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches 
        
           - Primary use case: Atomic updates (compare-and-swap)

Should be merged after #9332

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13919

How was this patch tested?

Clean CI: https://github.com/peterxcli/ozone/actions/runs/23552109067/job/68568728541

unit tests for affected components
rpc client integration test for newly introduced two Bucket API
s3 sdk test
robot test

…not exists" (If-None-Match) semantics.

…-create-if-not-exists

…nt tests and protocol definition.

…-create-if-not-exists

…ion handling - Implemented tests for `createKey` and `rewriteKey` methods to validate behavior when using the `EXPECTED_GEN_CREATE_IF_NOT_EXISTS` constant. - Added scenarios for key creation when the key is absent and when it already exists. - Enhanced the `rewriteFailsWhenKeyExists` test to cover cases for both committed and uncommitted keys. - Updated error handling to ensure correct responses for key existence checks.

…not exists" (If-None-Match) semantics.

…nt tests and protocol definition.

…ion handling - Implemented tests for `createKey` and `rewriteKey` methods to validate behavior when using the `EXPECTED_GEN_CREATE_IF_NOT_EXISTS` constant. - Added scenarios for key creation when the key is absent and when it already exists. - Enhanced the `rewriteFailsWhenKeyExists` test to cover cases for both committed and uncommitted keys. - Updated error handling to ensure correct responses for key existence checks.

jojochuang

from a quick glance it looks mostly reasonable.

jojochuang · 2026-03-07T01:00:13Z

hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java

+        .addAllMetadataGdpr(metadata)
+        .addAllTags(tags)
+        .setLatestVersionLocation(getLatestVersionLocation)
+        .setExpectedETag(expectedETag);


does it set setExpectedDataGeneration here?

no, please take a look at the design doc:

ozone/hadoop-hdds/docs/content/design/s3-conditional-requests.md

Lines 144 to 167 in 5e5243e

#### If-Match Implementation

To optimize performance and reduce latency, we avoid a pre-flight check (GetS3KeyDetails) and instead validate the ETag during the OM Write operation.

This requires adding an optional `expectedETag` field to `KeyArgs`. This approach optimizes the "happy path" (successful match) by removing an extra network round trip.

For failing requests, they still incur the cost of a write RPC and Raft log entry, but this is acceptable under optimistic concurrency control assumptions.

##### S3 Gateway Layer

1. Parse `If-Match: "<etag>"` header.

2. Populate `KeyArgs` with the parsed `expectedETag`.

3. Send the write request (CreateKey) to OM.

##### OM Create Phase

Validation is performed within the `validateAndUpdateCache` method to ensure atomicity within the Ratis state machine application.

1. **Locking**: The OM acquires the write lock for the bucket/key.

2. **Key Lookup**: Retrieve the existing key from `KeyTable`.

3. **Validation**:

- **Key Not Found**: If the key does not exist, throw `KEY_NOT_FOUND` (maps to S3 412).

- **No ETag Metadata**: If the existing key (e.g., uploaded via OFS) does not have an ETag property, throw `ETAG_NOT_AVAILABLE` (maps to S3 412). The precondition cannot be evaluated, so we must fail rather than silently proceed.

- **ETag Mismatch**: Compare `existingKey.ETag` with `expectedETag`. If they do not match, throw `ETAG_MISMATCH` (maps to S3 412).

4. **Extract Generation**: If ETag matches, extract `existingKey.updateID`.

5. **Create Open Key**: Create open key entry with `expectedDataGeneration = existingKey.updateID`.

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java

jojochuang · 2026-03-07T01:07:52Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

@@ -187,8 +188,18 @@ public Response put(
        throw newError(S3ErrorTable.NO_SUCH_BUCKET, bucketName, ex);
      } else if (ex.getResult() == ResultCodes.FILE_ALREADY_EXISTS) {
        throw newError(S3ErrorTable.NO_OVERWRITE, keyPath, ex);
+      } else if (ex.getResult() == ResultCodes.KEY_ALREADY_EXISTS) {
+        throw newError(PRECOND_FAILED, keyPath, ex);


would you like to consider having different error messages for different cases?

…-conditional-writes

peterxcli added 17 commits November 20, 2025 11:52

Recognizes expectedDataGeneration = -1L as a sentinel for "create if …

afa8b6c

…not exists" (If-None-Match) semantics.

correct the new result code position

7868a86

Merge remote-tracking branch 'upstream/master' into HDDS-13963-atomic…

d630699

…-create-if-not-exists

Merge remote-tracking branch 'upstream/master' into HDDS-13963-atomic…

f7ad505

…-create-if-not-exists

Merge remote-tracking branch 'upstream/master' into HDDS-13963-atomic…

2fdb6de

…-create-if-not-exists

HDDS-14070. Update key generation mismatch handling in Ozone RPC clie…

da492ce

…nt tests and protocol definition.

Introduce EXPECTED_GEN_CREATE_IF_NOT_EXISTS constant

0d4dfb0

Merge remote-tracking branch 'upstream/master' into HDDS-13963-atomic…

f238ade

…-create-if-not-exists

revert the new KEY_GENERATION_MISMATCH error code

4e9665c

Recognizes expectedDataGeneration = -1L as a sentinel for "create if …

817e33f

…not exists" (If-None-Match) semantics.

correct the new result code position

021747b

HDDS-14070. Update key generation mismatch handling in Ozone RPC clie…

0fd6730

…nt tests and protocol definition.

Introduce EXPECTED_GEN_CREATE_IF_NOT_EXISTS constant

c64beca

revert the new KEY_GENERATION_MISMATCH error code

6b33766

Implement S3 conditional write support

6c8a08e

peterxcli changed the title ~~Hdds 13919 conditional writes~~ HDDS-13919. S3 Conditional Writes (PutObject) Feb 24, 2026

jojochuang reviewed Mar 7, 2026

View reviewed changes

peterxcli added 9 commits March 13, 2026 01:09

Add aws kit tests for S3 conditional object put

57d7788

Add tests for S3 object put with conditional headers

6d85ee1

Merge branch 'master' into HDDS-13919-conditional-writes

c15cd6a

Merge branch 'master' into HDDS-13963-atomic-create-if-not-exists

88aa08e

Add ATOMIC_CREATE_IF_NOT_EXISTS version gate

a1898c0

suggestion

e5f5e3d

fix pmd

86489b6

Merge branch 'HDDS-13963-atomic-create-if-not-exists' into HDDS-13919…

9081e96

…-conditional-writes

fix pmd

d6395a9

peterxcli mentioned this pull request Mar 25, 2026

HDDS-13963. Atomic Create-If-Not-Exists #9332

Open

fix new smoke test

c10fb9e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-13919. S3 Conditional Writes (PutObject)#9815

HDDS-13919. S3 Conditional Writes (PutObject)#9815
peterxcli wants to merge 27 commits intoapache:masterfrom
peterxcli:HDDS-13919-conditional-writes

peterxcli commented Feb 24, 2026 •

edited

Loading

Uh oh!

jojochuang left a comment

Uh oh!

jojochuang Mar 7, 2026

Uh oh!

peterxcli Mar 25, 2026

Uh oh!

Uh oh!

jojochuang Mar 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	## Specification

	### AWS S3 Conditional Write Specification

	#### If-None-Match Header

	```
	If-None-Match: "*"
	```

	- Succeeds only if object does NOT exist
	- Returns `412 Precondition Failed` if object exists
	- Primary use case: Create-only semantics

	#### If-Match Header

	```
	If-Match: "<etag>"
	```

	- Succeeds only if object EXISTS and ETag matches
	- Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches
	- Primary use case: Atomic updates (compare-and-swap)

	#### If-Match Implementation

	To optimize performance and reduce latency, we avoid a pre-flight check (GetS3KeyDetails) and instead validate the ETag during the OM Write operation.
	This requires adding an optional `expectedETag` field to `KeyArgs`. This approach optimizes the "happy path" (successful match) by removing an extra network round trip.
	For failing requests, they still incur the cost of a write RPC and Raft log entry, but this is acceptable under optimistic concurrency control assumptions.

	##### S3 Gateway Layer

	1. Parse `If-Match: "<etag>"` header.
	2. Populate `KeyArgs` with the parsed `expectedETag`.
	3. Send the write request (CreateKey) to OM.

	##### OM Create Phase

	Validation is performed within the `validateAndUpdateCache` method to ensure atomicity within the Ratis state machine application.

	1. Locking: The OM acquires the write lock for the bucket/key.
	2. Key Lookup: Retrieve the existing key from `KeyTable`.
	3. Validation:
	- Key Not Found: If the key does not exist, throw `KEY_NOT_FOUND` (maps to S3 412).
	- No ETag Metadata: If the existing key (e.g., uploaded via OFS) does not have an ETag property, throw `ETAG_NOT_AVAILABLE` (maps to S3 412). The precondition cannot be evaluated, so we must fail rather than silently proceed.
	- ETag Mismatch: Compare `existingKey.ETag` with `expectedETag`. If they do not match, throw `ETAG_MISMATCH` (maps to S3 412).
	4. Extract Generation: If ETag matches, extract `existingKey.updateID`.
	5. Create Open Key: Create open key entry with `expectedDataGeneration = existingKey.updateID`.

Conversation

peterxcli commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

jojochuang Mar 7, 2026

Choose a reason for hiding this comment

Uh oh!

peterxcli Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jojochuang Mar 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

peterxcli commented Feb 24, 2026 •

edited

Loading