From 589d6c5c05bf76c107bf8ba88f92669b76cffd3e Mon Sep 17 00:00:00 2001 From: peterxcli Date: Thu, 20 Nov 2025 15:08:44 +0800 Subject: [PATCH 1/7] add design doc --- .../content/design/s3-conditional-requests.md | 149 ++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 hadoop-hdds/docs/content/design/s3-conditional-requests.md diff --git a/hadoop-hdds/docs/content/design/s3-conditional-requests.md b/hadoop-hdds/docs/content/design/s3-conditional-requests.md new file mode 100644 index 000000000000..1ce5d004fb8f --- /dev/null +++ b/hadoop-hdds/docs/content/design/s3-conditional-requests.md @@ -0,0 +1,149 @@ +--- +title: "S3 Conditional Requests" +summary: Design to support S3 conditional requests for atomic operations. +date: 2025-11-20 +jira: HDDS-13117 +status: draft +author: Chu Cheng Li +--- + +# S3 Conditional Requests Design + +## Background + +AWS S3 supports conditional requests using HTTP conditional headers, enabling atomic operations, cache optimization, and preventing race conditions. This includes: + +- **Conditional Writes** (PutObject): `If-Match` and `If-None-Match` headers for atomic operations +- **Conditional Reads** (GetObject, HeadObject): `If-Match`, `If-None-Match`, `If-Modified-Since`, `If-Unmodified-Since` for cache validation +- **Conditional Copy** (CopyObject): Conditions on both source and destination objects + +### Current State + +- HDDS-10656 implemented atomic rewrite using `expectedDataGeneration` +- OM HA uses single Raft group with single applier thread (Ratis StateMachineUpdater) +- S3 gateway doesn't expose conditional headers to OM layer + +## Use Cases + +### Conditional Writes +- **Atomic key rewrites**: Prevent race conditions when updating existing objects +- **Create-only semantics**: Prevent accidental overwrites (`If-None-Match: *`) +- **Optimistic locking**: Enable concurrent access with conflict detection +- **Leader election**: Implement distributed coordination using S3 as backing store + +### Conditional Reads +- **Bandwidth optimization**: Avoid downloading unchanged objects (304 Not Modified) +- **HTTP caching**: Support standard browser/CDN caching semantics +- **Conditional processing**: Only process objects that meet specific criteria + +### Conditional Copy +- **Atomic copy operations**: Copy only if source/destination meets specific conditions +- **Prevent overwrite**: Copy only if destination doesn't exist + +## AWS S3 Conditional Write + +### Specification + +#### If-None-Match Header + +``` +If-None-Match: "*" +``` + +- Succeeds only if object does NOT exist +- Returns `412 Precondition Failed` if object exists +- Primary use case: Create-only semantics + +#### If-Match Header + +``` +If-Match: "" +``` + +- Succeeds only if object EXISTS and ETag matches +- Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches +- Primary use case: Atomic updates (compare-and-swap) + +#### Restrictions + +- Cannot use both headers together in same request +- No additional charges for failed conditional requests + +### Implementation + +#### Architecture Overview + +#### If-None-Match Implementation + +##### S3 Gateway Layer + +1. Parse `If-None-Match: *`. +2. Set `existingKeyGeneration = -1`. +3. Call `RpcClient.rewriteKey()`. + +##### OM Create Phase + +1. Validate `expectedDataGeneration == -1`. +2. If key exists → throw `KEY_ALREADY_EXISTS`. +3. Store `-1` in open key metadata. + +##### OM Commit Phase + +1. Check `expectedDataGeneration == -1` from open key. +2. If key now exists (race condition) → throw `KEY_ALREADY_EXISTS`. +3. Commit key. + +##### Race Condition Handling + +Using `-1` ensures atomicity. If a concurrent write (Client B) commits between Client A's Create and Commit, Client A's commit fails the `-1` validation check (key now exists), preserving strict create-if-not-exists semantics. + +#### If-Match Implementation + +Leverages existing `expectedDataGeneration` from HDDS-10656: + +##### S3 Gateway Layer + +1. Parse `If-Match: ""` header +2. Look up existing key via `getS3KeyDetails()` +3. Validate ETag matches, else throw `PRECOND_FAILED` (412) +4. Extract `expectedGeneration` from existing key +5. Pass `expectedGeneration` to RpcClient + +##### OM Create Phase + +1. Receive `expectedDataGeneration` parameter +2. Look up current key and validate exists +3. Extract current key's `updateID` value +4. Create open key with `expectedDataGeneration = updateID` +5. Return stream to S3 gateway + +##### OM Commit Phase + +1. Read open key (contains `expectedDataGeneration`) +2. Read current committed key +3. Validate `current.updateID == openKey.expectedDataGeneration` +4. Commit if match, reject if mismatch (existing HDDS-10656 logic) + +#### Error Mapping + +| OM Error | S3 Status | S3 Error Code | Scenario | +|----------|-----------|---------------|----------| +| `KEY_ALREADY_EXISTS` | 412 | PreconditionFailed | If-None-Match failed | +| `KEY_NOT_FOUND` | 412 | PreconditionFailed | If-Match failed (key missing) | +| `ETAG_MISMATCH` | 412 | PreconditionFailed | If-Match failed (ETag mismatch) | +| `GENERATION_MISMATCH` | 412 | PreconditionFailed | If-Match failed (concurrent modification) | + +## AWS S3 Conditional Read + +TODO + +## AWS S3 Conditional Copy + +TODO + +## References + +- [AWS S3 Conditional Requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html) +- [RFC 7232 - HTTP Conditional Requests](https://tools.ietf.org/html/rfc7232) +- [HDDS-10656 - Atomic Rewrite Key](https://issues.apache.org/jira/browse/HDDS-10656) +- [Leader Election with S3 Conditional Writes](https://www.morling.dev/blog/leader-election-with-s3-conditional-writes/) From dc046a18beaae8ae76cee12e14f6b6d66d16184c Mon Sep 17 00:00:00 2001 From: peterxcli Date: Tue, 25 Nov 2025 00:25:05 +0800 Subject: [PATCH 2/7] WIP: Implement conditional write support in ObjectEndpoint for S3 API --- .../ozone/s3/endpoint/ObjectEndpoint.java | 81 +++- .../ozone/client/ClientProtocolStub.java | 2 +- .../s3/endpoint/TestConditionalWrite.java | 384 ++++++++++++++++++ .../ozone/s3/endpoint/TestListParts.java | 6 +- .../endpoint/TestMultipartUploadComplete.java | 2 +- .../endpoint/TestMultipartUploadWithCopy.java | 10 +- .../ozone/s3/endpoint/TestObjectGet.java | 4 +- .../ozone/s3/endpoint/TestObjectPut.java | 70 ++-- .../s3/endpoint/TestObjectTaggingDelete.java | 2 +- .../s3/endpoint/TestObjectTaggingGet.java | 2 +- .../s3/endpoint/TestObjectTaggingPut.java | 14 +- .../ozone/s3/endpoint/TestPartUpload.java | 14 +- .../s3/endpoint/TestPartUploadWithStream.java | 8 +- .../s3/endpoint/TestPermissionCheck.java | 4 +- .../s3/endpoint/TestUploadWithStream.java | 4 +- .../s3/metrics/TestS3GatewayMetrics.java | 30 +- 16 files changed, 550 insertions(+), 87 deletions(-) create mode 100644 hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestConditionalWrite.java diff --git a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java index b495ea346dc1..20b3df5c7853 100644 --- a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java +++ b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java @@ -227,6 +227,8 @@ public Response put( @PathParam("bucket") String bucketName, @PathParam("path") String keyPath, @HeaderParam("Content-Length") long length, + @HeaderParam("If-None-Match") String ifNoneMatch, + @HeaderParam("If-Match") String ifMatch, @QueryParam("partNumber") int partNumber, @QueryParam("uploadId") @DefaultValue("") String uploadID, @QueryParam("tagging") String taggingMarker, @@ -311,6 +313,49 @@ public Response put( return Response.ok().status(HttpStatus.SC_OK).build(); } + // Check conditional write headers + // If-None-Match and If-Match are mutually exclusive + if (ifNoneMatch != null && ifMatch != null) { + throw newError(INVALID_ARGUMENT, keyPath); + } + + Long expectedGeneration = null; + + // Handle If-None-Match: * (create only if key doesn't exist) + if (ifNoneMatch != null) { + if (!"*".equals(ifNoneMatch.trim())) { + throw newError(INVALID_ARGUMENT, keyPath); + } + expectedGeneration = -1L; + } + + // Handle If-Match: (update only if ETag matches) + if (ifMatch != null) { + try { + // Look up existing key to get its ETag and generation + OzoneKeyDetails existingKey = getClientProtocol().getS3KeyDetails(bucketName, keyPath); + String existingETag = existingKey.getMetadata().get(OzoneConsts.ETAG); + + // Strip quotes from the provided ETag + String providedETag = stripQuotes(ifMatch.trim()); + + // Compare ETags + if (existingETag == null || !existingETag.equals(providedETag)) { + // ETag doesn't match - return 412 Precondition Failed + throw newError(S3ErrorTable.PRECOND_FAILED, keyPath); + } + + // ETag matches - use the key's generation for optimistic locking + expectedGeneration = existingKey.getGeneration(); + } catch (OMException ex) { + if (ex.getResult() == ResultCodes.KEY_NOT_FOUND) { + // Key doesn't exist - return 404 Not Found for If-Match + throw newError(S3ErrorTable.NO_SUCH_KEY, keyPath, ex); + } + throw ex; + } + } + // Normal put object S3ChunkInputStreamInfo chunkInputStreamInfo = getS3ChunkInputStreamInfo(body, length, amzDecodedLength, keyPath); @@ -331,9 +376,21 @@ public Response put( eTag = keyWriteResult.getKey(); putLength = keyWriteResult.getValue(); } else { - try (OzoneOutputStream output = getClientProtocol().createKey( - volume.getName(), bucketName, keyPath, length, replicationConfig, - customMetadata, tags)) { + // Choose between rewriteKey (for If-Match and If-None-Match) or createKey (for normal put) + OzoneOutputStream output; + if (expectedGeneration != null) { + // If-Match: use rewriteKey with expectedDataGeneration for optimistic locking + output = getClientProtocol().rewriteKey( + volume.getName(), bucketName, keyPath, length, expectedGeneration, + replicationConfig, customMetadata); + } else { + // Normal create + output = getClientProtocol().createKey( + volume.getName(), bucketName, keyPath, length, replicationConfig, + customMetadata, tags); + } + + try { long metadataLatencyNs = getMetrics().updatePutKeyMetadataStats(startNanos); perf.appendMetaLatencyNanos(metadataLatencyNs); @@ -343,6 +400,8 @@ public Response put( digestInputStream.getMessageDigest().digest()) .toLowerCase(); output.getMetadata().put(ETAG, eTag); + } finally { + output.close(); } } getMetrics().incPutKeySuccessLength(putLength); @@ -375,6 +434,22 @@ public Response put( throw newError(S3ErrorTable.QUOTA_EXCEEDED, keyPath, ex); } else if (ex.getResult() == ResultCodes.BUCKET_NOT_FOUND) { throw newError(S3ErrorTable.NO_SUCH_BUCKET, bucketName, ex); + } else if (ex.getResult() == ResultCodes.KEY_ALREADY_EXISTS) { + // If this is a conditional write failure, return 412 Precondition Failed + if (ifNoneMatch != null && "*".equals(ifNoneMatch.trim())) { + throw newError(S3ErrorTable.PRECOND_FAILED, keyPath, ex); + } + throw newError(S3ErrorTable.NO_OVERWRITE, keyPath, ex); + } else if (ex.getResult() == ResultCodes.KEY_NOT_FOUND) { + // For If-Match, if key doesn't exist during rewrite, return 404 + // This can happen if key was deleted between the initial check and the rewrite + if (ifMatch != null) { + throw newError(S3ErrorTable.NO_SUCH_KEY, keyPath, ex); + } + throw newError(S3ErrorTable.NO_SUCH_KEY, keyPath, ex); + } else if (ex.getResult() == ResultCodes.KEY_GENERATION_MISMATCH) { + // Generation mismatch during If-Match - return 412 Precondition Failed + throw newError(S3ErrorTable.PRECOND_FAILED, keyPath, ex); } else if (ex.getResult() == ResultCodes.FILE_ALREADY_EXISTS) { throw newError(S3ErrorTable.NO_OVERWRITE, keyPath, ex); } diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java index 739babce1d06..8ccf6f9f3477 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java @@ -717,7 +717,7 @@ public ListSnapshotResponse listSnapshot( String prevSnapshot, int maxListResult) throws IOException { return null; } - + @Override public void deleteSnapshot(String volumeName, String bucketName, String snapshotName) diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestConditionalWrite.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestConditionalWrite.java new file mode 100644 index 000000000000..f8898a15f622 --- /dev/null +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestConditionalWrite.java @@ -0,0 +1,384 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.s3.endpoint; + +import static java.nio.charset.StandardCharsets.UTF_8; +import static org.apache.hadoop.ozone.s3.util.S3Consts.X_AMZ_CONTENT_SHA256; +import static org.apache.hadoop.ozone.s3.util.S3Utils.stripQuotes; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.when; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import javax.ws.rs.core.HttpHeaders; +import javax.ws.rs.core.Response; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; +import org.apache.hadoop.ozone.client.OzoneBucket; +import org.apache.hadoop.ozone.client.OzoneClient; +import org.apache.hadoop.ozone.client.OzoneClientStub; +import org.apache.hadoop.ozone.client.OzoneKeyDetails; +import org.apache.hadoop.ozone.s3.exception.OS3Exception; +import org.apache.hadoop.ozone.s3.exception.S3ErrorTable; +import org.apache.http.HttpStatus; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +/** + * Test conditional writes (If-Match, If-None-Match) for PutObject. + */ +class TestConditionalWrite { + private static final String BUCKET_NAME = "test-bucket"; + private static final String KEY_NAME = "test-key"; + private static final String CONTENT = "test content"; + + private OzoneClient clientStub; + private ObjectEndpoint objectEndpoint; + private HttpHeaders headers; + private OzoneBucket bucket; + + @BeforeEach + void setup() throws IOException { + OzoneConfiguration config = new OzoneConfiguration(); + + // Create client stub and object store stub + clientStub = new OzoneClientStub(); + + // Create bucket + clientStub.getObjectStore().createS3Bucket(BUCKET_NAME); + bucket = clientStub.getObjectStore().getS3Bucket(BUCKET_NAME); + + headers = mock(HttpHeaders.class); + when(headers.getHeaderString(X_AMZ_CONTENT_SHA256)).thenReturn("mockSignature"); + + // Create ObjectEndpoint and set client to OzoneClientStub + objectEndpoint = EndpointBuilder.newObjectEndpointBuilder() + .setClient(clientStub) + .setConfig(config) + .setHeaders(headers) + .build(); + + objectEndpoint = spy(objectEndpoint); + } + + /** + * Test If-None-Match: * succeeds when key doesn't exist. + */ + @Test + void testIfNoneMatchSuccessWhenKeyNotExists() throws Exception { + InputStream inputStream = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); + + Response response = objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + CONTENT.length(), + "*", // If-None-Match: * + null, // If-Match + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream + ); + + assertEquals(HttpStatus.SC_OK, response.getStatus()); + assertNotNull(response.getHeaderString("ETag")); + + // Verify key was created + OzoneKeyDetails keyDetails = bucket.getKey(KEY_NAME); + assertNotNull(keyDetails); + } + + /** + * Test If-None-Match: * fails when key already exists. + */ + @Test + void testIfNoneMatchFailsWhenKeyExists() throws Exception { + // First, create the key normally + InputStream inputStream1 = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); + Response response1 = objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + CONTENT.length(), + null, // If-None-Match + null, // If-Match + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream1 + ); + assertEquals(HttpStatus.SC_OK, response1.getStatus()); + + // Second, try to create the same key with If-None-Match: * + InputStream inputStream2 = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); + OS3Exception exception = assertThrows(OS3Exception.class, () -> { + objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + CONTENT.length(), + "*", // If-None-Match: * + null, // If-Match + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream2 + ); + }); + + assertEquals(HttpStatus.SC_PRECONDITION_FAILED, exception.getHttpCode()); + assertEquals(S3ErrorTable.PRECOND_FAILED.getCode(), exception.getCode()); + } + + /** + * Test If-Match succeeds when ETag matches. + */ + @Test + void testIfMatchSuccessWhenETagMatches() throws Exception { + // First, create the key normally + InputStream inputStream1 = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); + Response response1 = objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + CONTENT.length(), + null, // If-None-Match + null, // If-Match + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream1 + ); + assertEquals(HttpStatus.SC_OK, response1.getStatus()); + String etag = stripQuotes(response1.getHeaderString("ETag")); + + // Second, update the key with If-Match using the correct ETag + String newContent = "updated content"; + InputStream inputStream2 = new ByteArrayInputStream(newContent.getBytes(UTF_8)); + Response response2 = objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + newContent.length(), + null, // If-None-Match + "\"" + etag + "\"", // If-Match with quoted ETag + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream2 + ); + + assertEquals(HttpStatus.SC_OK, response2.getStatus()); + assertNotNull(response2.getHeaderString("ETag")); + } + + /** + * Test If-Match fails when key doesn't exist. + */ + @Test + void testIfMatchFailsWhenKeyNotExists() throws Exception { + String fakeEtag = "abc123"; + InputStream inputStream = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); + + OS3Exception exception = assertThrows(OS3Exception.class, () -> { + objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + CONTENT.length(), + null, // If-None-Match + "\"" + fakeEtag + "\"", // If-Match + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream + ); + }); + + assertEquals(HttpStatus.SC_NOT_FOUND, exception.getHttpCode()); + assertEquals(S3ErrorTable.NO_SUCH_KEY.getCode(), exception.getCode()); + } + + /** + * Test If-Match fails when ETag doesn't match. + */ + @Test + void testIfMatchFailsWhenETagMismatch() throws Exception { + // First, create the key normally + InputStream inputStream1 = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); + Response response1 = objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + CONTENT.length(), + null, // If-None-Match + null, // If-Match + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream1 + ); + assertEquals(HttpStatus.SC_OK, response1.getStatus()); + + // Second, try to update with wrong ETag + String wrongEtag = "wrongetag123"; + String newContent = "updated content"; + InputStream inputStream2 = new ByteArrayInputStream(newContent.getBytes(UTF_8)); + + OS3Exception exception = assertThrows(OS3Exception.class, () -> { + objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + newContent.length(), + null, // If-None-Match + "\"" + wrongEtag + "\"", // If-Match with wrong ETag + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream2 + ); + }); + + assertEquals(HttpStatus.SC_PRECONDITION_FAILED, exception.getHttpCode()); + assertEquals(S3ErrorTable.PRECOND_FAILED.getCode(), exception.getCode()); + } + + /** + * Test that If-Match and If-None-Match are mutually exclusive. + */ + @Test + void testIfMatchAndIfNoneMatchMutuallyExclusive() throws Exception { + InputStream inputStream = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); + + OS3Exception exception = assertThrows(OS3Exception.class, () -> { + objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + CONTENT.length(), + "*", // If-None-Match + "\"abc\"", // If-Match + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream + ); + }); + + assertEquals(HttpStatus.SC_BAD_REQUEST, exception.getHttpCode()); + assertEquals(S3ErrorTable.INVALID_ARGUMENT.getCode(), exception.getCode()); + } + + /** + * Test If-None-Match with invalid value (must be "*"). + */ + @Test + void testIfNoneMatchInvalidValue() throws Exception { + InputStream inputStream = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); + + OS3Exception exception = assertThrows(OS3Exception.class, () -> { + objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + CONTENT.length(), + "\"etag\"", // If-None-Match with invalid value (must be "*") + null, // If-Match + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream + ); + }); + + assertEquals(HttpStatus.SC_BAD_REQUEST, exception.getHttpCode()); + assertEquals(S3ErrorTable.INVALID_ARGUMENT.getCode(), exception.getCode()); + } + + /** + * Test generation mismatch detection - simulates concurrent modification. + * This test verifies that when a key is modified between the If-Match check + * and the actual rewrite, it returns 412 Precondition Failed. + */ + @Test + void testIfMatchGenerationMismatch() throws Exception { + // First, create the key normally + InputStream inputStream1 = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); + Response response1 = objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + CONTENT.length(), + null, // If-None-Match + null, // If-Match + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream1 + ); + assertEquals(HttpStatus.SC_OK, response1.getStatus()); + String etag = stripQuotes(response1.getHeaderString("ETag")); + + // Second, update the key normally (simulating concurrent modification) + String intermediateContent = "intermediate update"; + InputStream inputStream2 = new ByteArrayInputStream(intermediateContent.getBytes(UTF_8)); + Response response2 = objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + intermediateContent.length(), + null, // If-None-Match + null, // If-Match + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream2 + ); + assertEquals(HttpStatus.SC_OK, response2.getStatus()); + + // Third, try to update using the old ETag (should fail with generation mismatch) + String finalContent = "final update"; + InputStream inputStream3 = new ByteArrayInputStream(finalContent.getBytes(UTF_8)); + + OS3Exception exception = assertThrows(OS3Exception.class, () -> { + objectEndpoint.put( + BUCKET_NAME, + KEY_NAME, + finalContent.length(), + null, // If-None-Match + "\"" + etag + "\"", // If-Match with old ETag + 0, // partNumber + "", // uploadID + null, // taggingMarker + null, // aclMarker + inputStream3 + ); + }); + + // Should return 412 Precondition Failed (not 404) + assertEquals(HttpStatus.SC_PRECONDITION_FAILED, exception.getHttpCode()); + assertEquals(S3ErrorTable.PRECOND_FAILED.getCode(), exception.getCode()); + } +} + diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestListParts.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestListParts.java index 30be715b5305..0166cda24473 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestListParts.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestListParts.java @@ -76,17 +76,17 @@ public void setUp() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), 1, uploadID, null, null, body); + content.length(), null, null, 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), 2, uploadID, null, null, body); + content.length(), null, null, 2, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), 3, uploadID, null, null, body); + content.length(), null, null, 3, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); } diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadComplete.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadComplete.java index fde336f48079..abedf437cf79 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadComplete.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadComplete.java @@ -107,7 +107,7 @@ private Part uploadPart(String key, String uploadID, int partNumber, String ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); Response response = rest.put(OzoneConsts.S3_BUCKET, key, content.length(), - partNumber, uploadID, null, null, body); + null, null, partNumber, uploadID, null, null, body); assertEquals(200, response.getStatus()); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); Part part = new Part(); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadWithCopy.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadWithCopy.java index fd83523214ca..085591e4c241 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadWithCopy.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadWithCopy.java @@ -277,7 +277,7 @@ public enum CopyIfTimestampTestCase { MODIFIED_SINCE_FUTURE_TS_UNMODIFIED_SINCE_UNPARSABLE_TS( futureTimeStr, UNPARSABLE_TIME_STR, null); - + private final String modifiedTimestamp; private final String unmodifiedTimestamp; private final String errorCode; @@ -336,7 +336,7 @@ private Part uploadPart(String key, String uploadID, int partNumber, String ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); Response response = REST.put(OzoneConsts.S3_BUCKET, key, content.length(), - partNumber, uploadID, null, null, body); + null, null, partNumber, uploadID, null, null, body); assertEquals(200, response.getStatus()); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); Part part = new Part(); @@ -380,8 +380,8 @@ private Part uploadPartWithCopy(String key, String uploadID, int partNumber, setHeaders(additionalHeaders); ByteArrayInputStream body = new ByteArrayInputStream("".getBytes(UTF_8)); - Response response = REST.put(OzoneConsts.S3_BUCKET, key, 0, partNumber, - uploadID, null, null, body); + Response response = REST.put(OzoneConsts.S3_BUCKET, key, 0, null, null, + partNumber, uploadID, null, null, body); assertEquals(200, response.getStatus()); CopyPartResult result = (CopyPartResult) response.getEntity(); @@ -408,7 +408,7 @@ public void testUploadWithRangeCopyContentLength() OzoneConsts.S3_BUCKET + "/" + EXISTING_KEY); additionalHeaders.put(COPY_SOURCE_HEADER_RANGE, "bytes=0-3"); setHeaders(additionalHeaders); - REST.put(OzoneConsts.S3_BUCKET, KEY, 0, 1, uploadID, null, null, body); + REST.put(OzoneConsts.S3_BUCKET, KEY, 0, null, null, 1, uploadID, null, null, body); OzoneMultipartUploadPartListParts parts = CLIENT.getObjectStore().getS3Bucket(OzoneConsts.S3_BUCKET) .listParts(KEY, uploadID, 0, 100); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectGet.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectGet.java index 3e772f8b8bf7..b8d73b85c79d 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectGet.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectGet.java @@ -93,11 +93,11 @@ public void init() throws OS3Exception, IOException { ByteArrayInputStream body = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); rest.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - 1, null, null, null, body); + null, null, 1, null, null, null, body); // Create a key with object tags when(headers.getHeaderString(TAG_HEADER)).thenReturn("tag1=value1&tag2=value2"); rest.put(BUCKET_NAME, KEY_WITH_TAG, CONTENT.length(), - 1, null, null, null, body); + null, null, 1, null, null, null, body); context = mock(ContainerRequestContext.class); when(context.getUriInfo()).thenReturn(mock(UriInfo.class)); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectPut.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectPut.java index e5c34fb4e465..11722fb46893 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectPut.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectPut.java @@ -158,7 +158,7 @@ void testPutObject(int length, ReplicationConfig replication) throws IOException bucket.setReplicationConfig(replication); //WHEN - Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, length, 1, null, null, null, body); + Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, length, null, null, 1, null, null, null, body); //THEN assertEquals(200, response.getStatus()); @@ -185,7 +185,7 @@ void testPutObjectContentLength() throws IOException, OS3Exception { new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); long dataSize = CONTENT.length(); - objectEndpoint.put(BUCKET_NAME, KEY_NAME, dataSize, 0, null, null, null, body); + objectEndpoint.put(BUCKET_NAME, KEY_NAME, dataSize, null, null, 0, null, null, null, body); assertEquals(dataSize, getKeyDataSize()); } @@ -202,7 +202,7 @@ void testPutObjectContentLengthForStreaming() when(headers.getHeaderString(DECODED_CONTENT_LENGTH_HEADER)) .thenReturn("15"); - objectEndpoint.put(BUCKET_NAME, KEY_NAME, chunkedContent.length(), 0, null, null, + objectEndpoint.put(BUCKET_NAME, KEY_NAME, chunkedContent.length(), null, null, 0, null, null, null, new ByteArrayInputStream(chunkedContent.getBytes(UTF_8))); assertEquals(15, getKeyDataSize()); } @@ -218,7 +218,7 @@ public void testPutObjectWithTags() throws IOException, OS3Exception { objectEndpoint.setHeaders(headersWithTags); Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - 1, null, null, null, body); + null, null, 1, null, null, null, body); assertEquals(200, response.getStatus()); @@ -242,7 +242,7 @@ public void testPutObjectWithOnlyTagKey() throws Exception { try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - 1, null, null, null, body); + null, null, 1, null, null, null, body); fail("request with invalid query param should fail"); } catch (OS3Exception ex) { assertEquals(INVALID_TAG.getCode(), ex.getCode()); @@ -261,7 +261,7 @@ public void testPutObjectWithDuplicateTagKey() throws Exception { objectEndpoint.setHeaders(headersWithDuplicateTagKey); try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - 1, null, null, null, body); + null, null, 1, null, null, null, body); fail("request with duplicate tag key should fail"); } catch (OS3Exception ex) { assertEquals(INVALID_TAG.getCode(), ex.getCode()); @@ -281,7 +281,7 @@ public void testPutObjectWithLongTagKey() throws Exception { objectEndpoint.setHeaders(headersWithLongTagKey); try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - 1, null, null, null, body); + null, null, 1, null, null, null, body); fail("request with tag key exceeding the length limit should fail"); } catch (OS3Exception ex) { assertEquals(INVALID_TAG.getCode(), ex.getCode()); @@ -301,7 +301,7 @@ public void testPutObjectWithLongTagValue() throws Exception { when(headersWithLongTagValue.getHeaderString(TAG_HEADER)).thenReturn("tag1=" + longTagValue); try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - 1, null, null, null, body); + null, null, 1, null, null, null, body); fail("request with tag value exceeding the length limit should fail"); } catch (OS3Exception ex) { assertEquals(INVALID_TAG.getCode(), ex.getCode()); @@ -327,7 +327,7 @@ public void testPutObjectWithTooManyTags() throws Exception { objectEndpoint.setHeaders(headersWithTooManyTags); try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - 1, null, null, null, body); + null, null, 1, null, null, null, body); fail("request with number of tags exceeding limit should fail"); } catch (OS3Exception ex) { assertEquals(INVALID_TAG.getCode(), ex.getCode()); @@ -356,7 +356,7 @@ void testPutObjectWithSignedChunks() throws IOException, OS3Exception { //WHEN Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, - chunkedContent.length(), 1, null, null, null, + chunkedContent.length(), null, null, 1, null, null, null, new ByteArrayInputStream(chunkedContent.getBytes(UTF_8))); //THEN @@ -386,7 +386,7 @@ public void testPutObjectMessageDigestResetDuringException() throws OS3Exception new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT - .length(), 1, null, null, null, body); + .length(), null, null, 1, null, null, null, body); fail("Should throw IOException"); } catch (IOException ignored) { // Verify that the message digest is reset so that the instance can be reused for the @@ -411,7 +411,7 @@ void testCopyObject() throws IOException, OS3Exception { when(headers.getHeaderString(CUSTOM_METADATA_COPY_DIRECTIVE_HEADER)).thenReturn("COPY"); Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, - CONTENT.length(), 1, null, null, null, body); + CONTENT.length(), null, null, 1, null, null, null, body); OzoneInputStream ozoneInputStream = clientStub.getObjectStore() .getS3Bucket(BUCKET_NAME) @@ -436,7 +436,7 @@ void testCopyObject() throws IOException, OS3Exception { when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( BUCKET_NAME + "/" + urlEncode(KEY_NAME)); - response = objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), 1, + response = objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), null, null, 1, null, null, null, body); // Check destination key and response @@ -466,7 +466,7 @@ void testCopyObject() throws IOException, OS3Exception { metadataHeaders.remove(CUSTOM_METADATA_HEADER_PREFIX + "custom-key-1"); metadataHeaders.remove(CUSTOM_METADATA_HEADER_PREFIX + "custom-key-2"); - response = objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), 1, + response = objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), null, null, 1, null, null, null, body); ozoneInputStream = clientStub.getObjectStore().getS3Bucket(DEST_BUCKET_NAME) @@ -494,7 +494,7 @@ void testCopyObject() throws IOException, OS3Exception { // wrong copy metadata directive when(headers.getHeaderString(CUSTOM_METADATA_COPY_DIRECTIVE_HEADER)).thenReturn("INVALID"); OS3Exception e = assertThrows(OS3Exception.class, () -> objectEndpoint.put( - DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), 1, null, null, null, body), + DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), null, null, 1, null, null, null, body), "test copy object failed"); assertThat(e.getHttpCode()).isEqualTo(400); assertThat(e.getCode()).isEqualTo("InvalidArgument"); @@ -504,7 +504,7 @@ void testCopyObject() throws IOException, OS3Exception { // source and dest same e = assertThrows(OS3Exception.class, () -> objectEndpoint.put( - BUCKET_NAME, KEY_NAME, CONTENT.length(), 1, null, null, null, body), + BUCKET_NAME, KEY_NAME, CONTENT.length(), null, null, 1, null, null, null, body), "test copy object failed"); assertThat(e.getErrorMessage()).contains("This copy request is illegal"); @@ -512,28 +512,28 @@ void testCopyObject() throws IOException, OS3Exception { when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( NO_SUCH_BUCKET + "/" + urlEncode(KEY_NAME)); e = assertThrows(OS3Exception.class, () -> objectEndpoint.put(DEST_BUCKET_NAME, - DEST_KEY, CONTENT.length(), 1, null, null, null, body), "test copy object failed"); + DEST_KEY, CONTENT.length(), null, null, 1, null, null, null, body), "test copy object failed"); assertThat(e.getCode()).contains("NoSuchBucket"); // dest bucket not found when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( BUCKET_NAME + "/" + urlEncode(KEY_NAME)); e = assertThrows(OS3Exception.class, () -> objectEndpoint.put(NO_SUCH_BUCKET, - DEST_KEY, CONTENT.length(), 1, null, null, null, body), "test copy object failed"); + DEST_KEY, CONTENT.length(), null, null, 1, null, null, null, body), "test copy object failed"); assertThat(e.getCode()).contains("NoSuchBucket"); //Both source and dest bucket not found when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( NO_SUCH_BUCKET + "/" + urlEncode(KEY_NAME)); e = assertThrows(OS3Exception.class, () -> objectEndpoint.put(NO_SUCH_BUCKET, - DEST_KEY, CONTENT.length(), 1, null, null, null, body), "test copy object failed"); + DEST_KEY, CONTENT.length(), null, null, 1, null, null, null, body), "test copy object failed"); assertThat(e.getCode()).contains("NoSuchBucket"); // source key not found when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( BUCKET_NAME + "/" + urlEncode(NO_SUCH_BUCKET)); e = assertThrows(OS3Exception.class, () -> objectEndpoint.put( - "nonexistent", KEY_NAME, CONTENT.length(), 1, null, null, null, body), + "nonexistent", KEY_NAME, CONTENT.length(), null, null, 1, null, null, null, body), "test copy object failed"); assertThat(e.getCode()).contains("NoSuchBucket"); } @@ -545,7 +545,7 @@ public void testCopyObjectMessageDigestResetDuringException() throws IOException new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, - CONTENT.length(), 1, null, null, null, body); + CONTENT.length(), null, null, 1, null, null, null, body); OzoneInputStream ozoneInputStream = clientStub.getObjectStore() .getS3Bucket(BUCKET_NAME) @@ -572,7 +572,7 @@ public void testCopyObjectMessageDigestResetDuringException() throws IOException BUCKET_NAME + "/" + urlEncode(KEY_NAME)); try { - objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), 1, + objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), null, null, 1, null, null, null, body); fail("Should throw IOException"); } catch (IOException ignored) { @@ -596,7 +596,7 @@ public void testCopyObjectWithTags() throws IOException, OS3Exception { String sourceKeyName = "sourceKey"; Response putResponse = objectEndpoint.put(BUCKET_NAME, sourceKeyName, - CONTENT.length(), 1, null, null, null, body); + CONTENT.length(), null, null, 1, null, null, null, body); OzoneKeyDetails keyDetails = clientStub.getObjectStore().getS3Bucket(BUCKET_NAME).getKey(sourceKeyName); @@ -614,7 +614,8 @@ public void testCopyObjectWithTags() throws IOException, OS3Exception { BUCKET_NAME + "/" + urlEncode(sourceKeyName)); objectEndpoint.setHeaders(headersForCopy); - Response copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), 1, null, null, null, body); + Response copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), null, null, 1, null, null, + null, body); OzoneKeyDetails destKeyDetails = clientStub.getObjectStore() .getS3Bucket(DEST_BUCKET_NAME).getKey(destKey); @@ -633,7 +634,8 @@ public void testCopyObjectWithTags() throws IOException, OS3Exception { // With x-amz-tagging-directive = COPY with a different x-amz-tagging when(headersForCopy.getHeaderString(TAG_HEADER)).thenReturn("tag3=value3"); - copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), 1, null, null, null, body); + copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), null, null, 1, null, null, null, + body); assertEquals(200, copyResponse.getStatus()); destKeyDetails = clientStub.getObjectStore() @@ -648,7 +650,8 @@ public void testCopyObjectWithTags() throws IOException, OS3Exception { // Copy object with x-amz-tagging-directive = REPLACE when(headersForCopy.getHeaderString(TAG_DIRECTIVE_HEADER)).thenReturn("REPLACE"); - copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), 1, null, null, null, body); + copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), null, null, 1, null, null, null, + body); assertEquals(200, copyResponse.getStatus()); destKeyDetails = clientStub.getObjectStore() @@ -670,7 +673,7 @@ public void testCopyObjectWithInvalidTagCopyDirective() throws Exception { HttpHeaders headersForCopy = Mockito.mock(HttpHeaders.class); when(headersForCopy.getHeaderString(TAG_DIRECTIVE_HEADER)).thenReturn("INVALID"); try { - objectEndpoint.put(DEST_BUCKET_NAME, "somekey", CONTENT.length(), 1, null, null, null, body); + objectEndpoint.put(DEST_BUCKET_NAME, "somekey", CONTENT.length(), null, null, 1, null, null, null, body); } catch (OS3Exception ex) { assertEquals(INVALID_ARGUMENT.getCode(), ex.getCode()); assertThat(ex.getErrorMessage()).contains("The tagging copy directive specified is invalid"); @@ -685,7 +688,7 @@ void testInvalidStorageType() { when(headers.getHeaderString(STORAGE_CLASS_HEADER)).thenReturn("random"); OS3Exception e = assertThrows(OS3Exception.class, () -> objectEndpoint.put( - BUCKET_NAME, KEY_NAME, CONTENT.length(), 1, null, null, null, body)); + BUCKET_NAME, KEY_NAME, CONTENT.length(), null, null, 1, null, null, null, body)); assertEquals(S3ErrorTable.INVALID_STORAGE_CLASS.getErrorMessage(), e.getErrorMessage()); assertEquals("random", e.getResource()); @@ -698,7 +701,7 @@ void testEmptyStorageType() throws IOException, OS3Exception { when(headers.getHeaderString(STORAGE_CLASS_HEADER)).thenReturn(""); objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT - .length(), 1, null, null, null, body); + .length(), null, null, 1, null, null, null, body); OzoneKeyDetails key = clientStub.getObjectStore().getS3Bucket(BUCKET_NAME) .getKey(KEY_NAME); @@ -717,7 +720,7 @@ void testDirectoryCreation() throws IOException, // WHEN try (Response response = objectEndpoint.put(fsoBucket.getName(), path, - 0L, 0, "", null, null, null)) { + 0L, null, null, 0, "", null, null, null)) { assertEquals(HttpStatus.SC_OK, response.getStatus()); } @@ -732,12 +735,12 @@ void testDirectoryCreationOverFile() throws IOException, OS3Exception { final String path = "key"; final ByteArrayInputStream body = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); - objectEndpoint.put(FSO_BUCKET_NAME, path, CONTENT.length(), 0, "", null, null, body); + objectEndpoint.put(FSO_BUCKET_NAME, path, CONTENT.length(), null, null, 0, "", null, null, body); // WHEN final OS3Exception exception = assertThrows(OS3Exception.class, () -> objectEndpoint - .put(FSO_BUCKET_NAME, path + "/", 0, 0, "", null, null, null) + .put(FSO_BUCKET_NAME, path + "/", 0, null, null, 0, "", null, null, null) .close()); // THEN @@ -753,7 +756,8 @@ public void testPutEmptyObject() throws IOException, OS3Exception { ByteArrayInputStream body = new ByteArrayInputStream(emptyString.getBytes(UTF_8)); objectEndpoint.setHeaders(headersWithTags); - Response putResponse = objectEndpoint.put(BUCKET_NAME, KEY_NAME, emptyString.length(), 1, null, null, null, body); + Response putResponse = objectEndpoint.put(BUCKET_NAME, KEY_NAME, emptyString.length(), null, null, 1, null, null, + null, body); assertEquals(200, putResponse.getStatus()); OzoneKeyDetails keyDetails = clientStub.getObjectStore().getS3Bucket(BUCKET_NAME).getKey(KEY_NAME); assertEquals(0, keyDetails.getDataSize()); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingDelete.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingDelete.java index 81a260ff4f21..8c249566b290 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingDelete.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingDelete.java @@ -84,7 +84,7 @@ public void init() throws OS3Exception, IOException { Mockito.when(headers.getHeaderString(X_AMZ_CONTENT_SHA256)) .thenReturn("mockSignature"); rest.put(BUCKET_NAME, KEY_WITH_TAG, CONTENT.length(), - 1, null, null, null, body); + null, null, 1, null, null, null, body); ContainerRequestContext context = Mockito.mock(ContainerRequestContext.class); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingGet.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingGet.java index c4eb4c25ff87..6e3d1823e4e2 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingGet.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingGet.java @@ -75,7 +75,7 @@ public void init() throws OS3Exception, IOException { // Create a key with object tags Mockito.when(headers.getHeaderString(TAG_HEADER)).thenReturn("tag1=value1&tag2=value2"); rest.put(BUCKET_NAME, KEY_WITH_TAG, CONTENT.length(), - 1, null, null, null, body); + null, null, 1, null, null, null, body); ContainerRequestContext context = Mockito.mock(ContainerRequestContext.class); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingPut.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingPut.java index 02b71e8772c4..5f56aff66a70 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingPut.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingPut.java @@ -89,13 +89,13 @@ void setup() throws IOException, OS3Exception { ByteArrayInputStream body = new ByteArrayInputStream("".getBytes(UTF_8)); - objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, 1, null, null, null, body); + objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, null, null, 1, null, null, null, body); } @Test public void testPutObjectTaggingWithEmptyBody() throws Exception { try { - objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, 1, null, "", null, + objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, null, null, 1, null, "", null, null); fail(); } catch (OS3Exception ex) { @@ -106,7 +106,7 @@ public void testPutObjectTaggingWithEmptyBody() throws Exception { @Test public void testPutValidObjectTagging() throws Exception { - assertEquals(HTTP_OK, objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, 1, null, + assertEquals(HTTP_OK, objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, null, null, 1, null, "", null, twoTags()).getStatus()); OzoneKeyDetails keyDetails = clientStub.getObjectStore().getS3Bucket(BUCKET_NAME).getKey(KEY_NAME); @@ -128,7 +128,7 @@ public void testPutInvalidObjectTagging() throws Exception { private void testInvalidObjectTagging(Supplier inputStream, int expectedHttpCode, String expectedErrorCode) throws Exception { try { - objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, 1, null, "", null, + objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, null, null, 1, null, "", null, inputStream.get()); fail("Expected an OS3Exception to be thrown"); } catch (OS3Exception ex) { @@ -140,7 +140,7 @@ private void testInvalidObjectTagging(Supplier inputStream, @Test public void testPutObjectTaggingNoKeyFound() throws Exception { try { - objectEndpoint.put(BUCKET_NAME, "nonexistent", 0, 1, + objectEndpoint.put(BUCKET_NAME, "nonexistent", 0, null, null, 1, null, "", null, twoTags()); fail("Expected an OS3Exception to be thrown"); } catch (OS3Exception ex) { @@ -152,7 +152,7 @@ public void testPutObjectTaggingNoKeyFound() throws Exception { @Test public void testPutObjectTaggingNoBucketFound() throws Exception { try { - objectEndpoint.put("nonexistent", "nonexistent", 0, 1, + objectEndpoint.put("nonexistent", "nonexistent", 0, null, null, 1, null, "", null, twoTags()); fail("Expected an OS3Exception to be thrown"); } catch (OS3Exception ex) { @@ -184,7 +184,7 @@ public void testPutObjectTaggingNotImplemented() throws Exception { ResultCodes.NOT_SUPPORTED_OPERATION)).when(mockBucket).putObjectTagging("dir/", twoTagsMap); try { - endpoint.put("fsoBucket", "dir/", 0, 1, null, "", + endpoint.put("fsoBucket", "dir/", 0, null, null, 1, null, "", null, twoTags()); fail("Expected an OS3Exception to be thrown"); } catch (OS3Exception ex) { diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUpload.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUpload.java index 4981069528a8..dd206f06bae2 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUpload.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUpload.java @@ -97,7 +97,7 @@ public void testPartUpload() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), 1, uploadID, null, null, body); + content.length(), null, null, 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); @@ -119,7 +119,7 @@ public void testPartUploadWithOverride() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), 1, uploadID, null, null, body); + content.length(), null, null, 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); @@ -128,7 +128,7 @@ public void testPartUploadWithOverride() throws Exception { // Upload part again with same part Number, the ETag should be changed. content = "Multipart Upload Changed"; response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), 1, uploadID, null, null, body); + content.length(), null, null, 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); assertNotEquals(eTag, response.getHeaderString(OzoneConsts.ETAG)); @@ -140,7 +140,7 @@ public void testPartUploadWithIncorrectUploadID() throws Exception { String content = "Multipart Upload With Incorrect uploadID"; ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); - rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, content.length(), 1, + rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, content.length(), null, null, 1, "random", null, null, body); }); assertEquals("NoSuchUpload", ex.getCode()); @@ -176,7 +176,7 @@ public void testPartUploadStreamContentLength() String uploadID = multipartUploadInitiateResponse.getUploadID(); long contentLength = chunkedContent.length(); - objectEndpoint.put(OzoneConsts.S3_BUCKET, keyName, contentLength, 1, + objectEndpoint.put(OzoneConsts.S3_BUCKET, keyName, contentLength, null, null, 1, uploadID, null, null, new ByteArrayInputStream(chunkedContent.getBytes(UTF_8))); assertContentLength(uploadID, keyName, 15); } @@ -200,7 +200,7 @@ public void testPartUploadContentLength() throws IOException, OS3Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); rest.put(OzoneConsts.S3_BUCKET, keyName, - contentLength, 1, uploadID, null, null, body); + contentLength, null, null, 1, uploadID, null, null, body); assertContentLength(uploadID, keyName, content.length()); } @@ -245,7 +245,7 @@ public void testPartUploadMessageDigestResetDuringException() throws IOException new ByteArrayInputStream(content.getBytes(UTF_8)); try { objectEndpoint.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), 1, uploadID, null, null, body); + content.length(), null, null, 1, uploadID, null, null, body); fail("Should throw IOException"); } catch (IOException ignored) { // Verify that the message digest is reset so that the instance can be reused for the diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUploadWithStream.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUploadWithStream.java index 4b2d8a49efb9..ccdad9ada3b5 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUploadWithStream.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUploadWithStream.java @@ -96,7 +96,7 @@ public void testPartUpload() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); response = rest.put(S3BUCKET, S3KEY, - content.length(), 1, uploadID, null, null, body); + content.length(), null, null, 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); @@ -117,7 +117,7 @@ public void testPartUploadWithOverride() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); response = rest.put(S3BUCKET, S3KEY, - content.length(), 1, uploadID, null, null, body); + content.length(), null, null, 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); @@ -126,7 +126,7 @@ public void testPartUploadWithOverride() throws Exception { // Upload part again with same part Number, the ETag should be changed. content = "Multipart Upload Changed"; response = rest.put(S3BUCKET, S3KEY, - content.length(), 1, uploadID, null, null, body); + content.length(), null, null, 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); assertNotEquals(eTag, response.getHeaderString(OzoneConsts.ETAG)); @@ -138,7 +138,7 @@ public void testPartUploadWithIncorrectUploadID() throws Exception { String content = "Multipart Upload With Incorrect uploadID"; ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); - rest.put(S3BUCKET, S3KEY, content.length(), 1, + rest.put(S3BUCKET, S3KEY, content.length(), null, null, 1, "random", null, null, body); }); assertEquals("NoSuchUpload", ex.getCode()); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPermissionCheck.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPermissionCheck.java index 81f6853bf73f..f4768552d947 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPermissionCheck.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPermissionCheck.java @@ -277,7 +277,7 @@ public void testPutKey() throws IOException { .build(); OS3Exception e = assertThrows(OS3Exception.class, () -> objectEndpoint.put( - "bucketName", "keyPath", 1024, 0, null, null, null, + "bucketName", "keyPath", 1024, null, null, 0, null, null, null, new ByteArrayInputStream(new byte[]{}))); assertEquals(HTTP_FORBIDDEN, e.getHttpCode()); } @@ -340,7 +340,7 @@ public void testObjectTagging() throws Exception { InputStream tagInput = new ByteArrayInputStream(xml.getBytes(UTF_8)); OS3Exception e = assertThrows(OS3Exception.class, () -> - objectEndpoint.put("bucketName", "keyPath", 0, 1, + objectEndpoint.put("bucketName", "keyPath", 0, null, null, 1, null, "", null, tagInput)); assertEquals(HTTP_FORBIDDEN, e.getHttpCode()); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestUploadWithStream.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestUploadWithStream.java index 7ed2c488c444..adefc07b2e63 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestUploadWithStream.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestUploadWithStream.java @@ -103,7 +103,7 @@ public void testUpload() throws Exception { byte[] keyContent = S3_COPY_EXISTING_KEY_CONTENT.getBytes(UTF_8); ByteArrayInputStream body = new ByteArrayInputStream(keyContent); - Response response = rest.put(S3BUCKET, S3KEY, 0, 0, null, null, null, body); + Response response = rest.put(S3BUCKET, S3KEY, 0, null, null, 0, null, null, null, body); assertEquals(200, response.getStatus()); } @@ -137,7 +137,7 @@ public void testUploadWithCopy() throws Exception { .forEach((k, v) -> when(headers.getHeaderString(k)).thenReturn(v)); rest.setHeaders(headers); - Response response = rest.put(S3BUCKET, S3KEY, 0, 0, null, null, null, null); + Response response = rest.put(S3BUCKET, S3KEY, 0, null, null, 0, null, null, null, null); assertEquals(200, response.getStatus()); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/metrics/TestS3GatewayMetrics.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/metrics/TestS3GatewayMetrics.java index 600df053c4ec..91df86eab68d 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/metrics/TestS3GatewayMetrics.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/metrics/TestS3GatewayMetrics.java @@ -310,7 +310,7 @@ public void testCreateKeySuccess() throws Exception { new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); // Create the file keyEndpoint.put(bucketName, keyName, CONTENT - .length(), 1, null, null, null, body); + .length(), null, null, 1, null, null, null, body); body.close(); long curMetric = metrics.getCreateKeySuccess(); assertEquals(1L, curMetric - oriMetric); @@ -322,7 +322,7 @@ public void testCreateKeyFailure() throws Exception { // Create the file in a bucket that does not exist OS3Exception e = assertThrows(OS3Exception.class, () -> keyEndpoint.put( - "unknownBucket", keyName, CONTENT.length(), 1, null, null, + "unknownBucket", keyName, CONTENT.length(), null, null, 1, null, null, null, null)); assertEquals(S3ErrorTable.NO_SUCH_BUCKET.getCode(), e.getCode()); long curMetric = metrics.getCreateKeyFailure(); @@ -358,7 +358,7 @@ public void testGetKeySuccess() throws Exception { new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); // Create the file keyEndpoint.put(bucketName, keyName, CONTENT - .length(), 1, null, null, null, body); + .length(), null, null, 1, null, null, null, body); // GET the key from the bucket Response response = keyEndpoint.get(bucketName, keyName, 0, null, 0, null, null); StreamingOutput stream = (StreamingOutput) response.getEntity(); @@ -465,7 +465,7 @@ public void testCreateMultipartKeySuccess() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); keyEndpoint.put(bucketName, keyName, CONTENT.length(), - 1, uploadID, null, null, body); + null, null, 1, uploadID, null, null, body); long curMetric = metrics.getCreateMultipartKeySuccess(); assertEquals(1L, curMetric - oriMetric); } @@ -474,7 +474,7 @@ public void testCreateMultipartKeySuccess() throws Exception { public void testCreateMultipartKeyFailure() throws Exception { long oriMetric = metrics.getCreateMultipartKeyFailure(); OS3Exception e = assertThrows(OS3Exception.class, () -> keyEndpoint.put( - bucketName, keyName, CONTENT.length(), 1, "randomId", null, null, null)); + bucketName, keyName, CONTENT.length(), null, null, 1, "randomId", null, null, null)); assertEquals(S3ErrorTable.NO_SUCH_UPLOAD.getCode(), e.getCode()); long curMetric = metrics.getCreateMultipartKeyFailure(); assertEquals(1L, curMetric - oriMetric); @@ -521,13 +521,13 @@ public void testCopyObject() throws Exception { new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); keyEndpoint.put(bucketName, keyName, - CONTENT.length(), 1, null, null, null, body); + CONTENT.length(), null, null, 1, null, null, null, body); // Add copy header, and then call put when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( bucketName + "/" + urlEncode(keyName)); - keyEndpoint.put(destBucket, destKey, CONTENT.length(), 1, + keyEndpoint.put(destBucket, destKey, CONTENT.length(), null, null, 1, null, null, null, body); long curMetric = metrics.getCopyObjectSuccess(); assertEquals(1L, curMetric - oriMetric); @@ -537,7 +537,7 @@ public void testCopyObject() throws Exception { // source and dest same when(headers.getHeaderString(STORAGE_CLASS_HEADER)).thenReturn(""); OS3Exception e = assertThrows(OS3Exception.class, () -> keyEndpoint.put( - bucketName, keyName, CONTENT.length(), 1, null, null, null, body), + bucketName, keyName, CONTENT.length(), null, null, 1, null, null, null, body), "Test for CopyObjectMetric failed"); assertThat(e.getErrorMessage()).contains("This copy request is illegal"); curMetric = metrics.getCopyObjectFailure(); @@ -552,11 +552,11 @@ public void testPutObjectTaggingSuccess() throws Exception { new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); // Create the file keyEndpoint.put(bucketName, keyName, CONTENT - .length(), 1, null, null, null, body); + .length(), null, null, 1, null, null, null, body); body.close(); // Put object tagging - keyEndpoint.put(bucketName, keyName, 0, 1, null, "", null, getPutTaggingBody()); + keyEndpoint.put(bucketName, keyName, 0, null, null, 1, null, "", null, getPutTaggingBody()); long curMetric = metrics.getPutObjectTaggingSuccess(); assertEquals(1L, curMetric - oriMetric); @@ -568,7 +568,7 @@ public void testPutObjectTaggingFailure() throws Exception { // Put object tagging for nonexistent key OS3Exception ex = assertThrows(OS3Exception.class, () -> - keyEndpoint.put(bucketName, "nonexistent", 0, 1, null, "", + keyEndpoint.put(bucketName, "nonexistent", 0, null, null, 1, null, "", null, getPutTaggingBody()) ); assertEquals(S3ErrorTable.NO_SUCH_KEY.getCode(), ex.getCode()); @@ -585,11 +585,11 @@ public void testGetObjectTaggingSuccess() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); keyEndpoint.put(bucketName, keyName, CONTENT - .length(), 1, null, null, null, body); + .length(), null, null, 1, null, null, null, body); body.close(); // Put object tagging - keyEndpoint.put(bucketName, keyName, 0, 1, null, "", null, getPutTaggingBody()); + keyEndpoint.put(bucketName, keyName, 0, null, null, 1, null, "", null, getPutTaggingBody()); // Get object tagging keyEndpoint.get(bucketName, keyName, 0, @@ -620,11 +620,11 @@ public void testDeleteObjectTaggingSuccess() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); keyEndpoint.put(bucketName, keyName, CONTENT - .length(), 1, null, null, null, body); + .length(), null, null, 1, null, null, null, body); body.close(); // Put object tagging - keyEndpoint.put(bucketName, keyName, 0, 1, null, "", null, getPutTaggingBody()); + keyEndpoint.put(bucketName, keyName, 0, null, null, 1, null, "", null, getPutTaggingBody()); // Delete object tagging keyEndpoint.delete(bucketName, keyName, null, ""); From f7b886f2608138e74e327551231de262a1a93095 Mon Sep 17 00:00:00 2001 From: peterxcli Date: Sun, 30 Nov 2025 17:48:59 +0800 Subject: [PATCH 3/7] Refine design and specify If-Match ETag validation in OM write path (no pre-flight RPC) - Add If-Match implementation: validate ETag in OM validateAndUpdateCache, avoiding GetS3KeyDetails pre-flight check to optimize happy path - Document If-None-Match using EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS=-1 constant for atomic create-if-not-exists semantics - Reorganize spec sections: separate Write/Read/Copy specifications - Clarify OM validation logic: locking, key lookup, ETag comparison, error cases - Update error mapping: add PRECONDITION_FAILED for missing ETag scenarios - Add HDDS-13963 reference for Create-If-Not-Exists capability --- .../content/design/s3-conditional-requests.md | 107 +++++++++++------- 1 file changed, 69 insertions(+), 38 deletions(-) diff --git a/hadoop-hdds/docs/content/design/s3-conditional-requests.md b/hadoop-hdds/docs/content/design/s3-conditional-requests.md index 1ce5d004fb8f..1901649ce3ec 100644 --- a/hadoop-hdds/docs/content/design/s3-conditional-requests.md +++ b/hadoop-hdds/docs/content/design/s3-conditional-requests.md @@ -26,23 +26,26 @@ AWS S3 supports conditional requests using HTTP conditional headers, enabling at ## Use Cases ### Conditional Writes + - **Atomic key rewrites**: Prevent race conditions when updating existing objects - **Create-only semantics**: Prevent accidental overwrites (`If-None-Match: *`) - **Optimistic locking**: Enable concurrent access with conflict detection - **Leader election**: Implement distributed coordination using S3 as backing store ### Conditional Reads + - **Bandwidth optimization**: Avoid downloading unchanged objects (304 Not Modified) - **HTTP caching**: Support standard browser/CDN caching semantics - **Conditional processing**: Only process objects that meet specific criteria ### Conditional Copy + - **Atomic copy operations**: Copy only if source/destination meets specific conditions - **Prevent overwrite**: Copy only if destination doesn't exist -## AWS S3 Conditional Write +## Specification -### Specification +### AWS S3 Conditional Write Specification #### If-None-Match Header @@ -69,75 +72,101 @@ If-Match: "" - Cannot use both headers together in same request - No additional charges for failed conditional requests -### Implementation +### AWS S3 Conditional Read Specification + +TODO + +### AWS S3 Conditional Copy Specification + +TODO -#### Architecture Overview +## Implementation + +### AWS S3 Conditional Write Implementation + +The implementation aims to minimize Redundant RPCs (RTT) while ensuring strict atomicity for conditional operations. + +- **If-None-Match** utilizes the atomic "Create-If-Not-Exists" capability ([HDDS-13963](https://issues.apache.org/jira/browse/HDDS-13963 "null")). +- **If-Match** optimizes the happy path by pushing ETag validation directly into the Ozone Manager's write path, avoiding preliminary read operations. #### If-None-Match Implementation +This implementation ensures strict create-only semantics by utilizing a specific generation ID marker. + +In `OzoneConsts.java`, add the `-1` as a constant for readability: +```java +/** + * Special value for expectedDataGeneration to indicate "Create-If-Not-Exists" semantics. + * When used with If-None-Match conditional requests, this ensures atomicity: + * if a concurrent write commits between Create and Commit phases, the commit + * fails the validation check, preserving strict create-if-not-exists semantics. + */ +public static final long EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS = -1L; +``` + ##### S3 Gateway Layer 1. Parse `If-None-Match: *`. -2. Set `existingKeyGeneration = -1`. +2. Set `existingKeyGeneration = OzoneConsts.EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS`. 3. Call `RpcClient.rewriteKey()`. ##### OM Create Phase -1. Validate `expectedDataGeneration == -1`. -2. If key exists → throw `KEY_ALREADY_EXISTS`. -3. Store `-1` in open key metadata. +1. OM receives request with `expectedDataGeneration == OzoneConsts.EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS`. +2. **Pre-check**: If key is already in the OpenKeyTable or KeyTable, throw `KEY_ALREADY_EXISTS`. +3. If not exists, proceed to create the open key entry. -##### OM Commit Phase +##### OM Commit Phase (Atomicity) -1. Check `expectedDataGeneration == -1` from open key. -2. If key now exists (race condition) → throw `KEY_ALREADY_EXISTS`. -3. Commit key. +1. During the commit phase (or strict atomic create), the OM validates that the key still does not exist. +2. If a concurrent client created the key between the Create and Commit phases, the transaction fails with `KEY_ALREADY_EXISTS`. ##### Race Condition Handling -Using `-1` ensures atomicity. If a concurrent write (Client B) commits between Client A's Create and Commit, Client A's commit fails the `-1` validation check (key now exists), preserving strict create-if-not-exists semantics. +Using `OzoneConsts.EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS = -1` ensures atomicity. If a concurrent write (Client B) commits between Client A's Create and Commit, +Client A's commit fails the `CREATE IF NOT EXISTS` validation check, preserving strict create-if-not-exists semantics. #### If-Match Implementation -Leverages existing `expectedDataGeneration` from HDDS-10656: +To optimize performance and reduce latency, we avoid a pre-flight check (GetS3KeyDetails) and instead validate the ETag during the OM Write operation. +This requires adding an optional `expectedETag` field to `KeyArgs`. This approach optimizes the "happy path" (successful match) by removing an extra network round trip. +For failing requests, they still incur the cost of a write RPC and Raft log entry, but this is acceptable under optimistic concurrency control assumptions. ##### S3 Gateway Layer -1. Parse `If-Match: ""` header -2. Look up existing key via `getS3KeyDetails()` -3. Validate ETag matches, else throw `PRECOND_FAILED` (412) -4. Extract `expectedGeneration` from existing key -5. Pass `expectedGeneration` to RpcClient +1. Parse `If-Match: ""` header. +3. Populate `KeyArgs` with the parsed `expectedETag`. +4. Send the write request (CreateKey/OpenKey) to OM. -##### OM Create Phase +##### OM Layer (Validation Logic) + +Validation is performed within the `validateAndUpdateCache` method to ensure atomicity within the Ratis state machine application. -1. Receive `expectedDataGeneration` parameter -2. Look up current key and validate exists -3. Extract current key's `updateID` value -4. Create open key with `expectedDataGeneration = updateID` -5. Return stream to S3 gateway +1. **Locking**: The OM acquires the write lock for the bucket/key. +2. **Key Lookup**: Retrieve the existing key from `KeyTable`. +3. **Validation**: -##### OM Commit Phase + - **Key Not Found**: If the key does not exist, throw `KEY_NOT_FOUND` (maps to S3 412). + - **No ETag Metadata**: If the existing key (e.g., uploaded via OFS) does not have an ETag property, validation fails. We do **not** calculate ETag on the spot to avoid performance overhead on the applier thread. Throws `PRECONDITION_FAILED`. + - **ETag Mismatch**: Compare `existingKey.ETag` with `expectedETag`. If they do not match, throw `PRECONDITION_FAILED` (maps to S3 412). -1. Read open key (contains `expectedDataGeneration`) -2. Read current committed key -3. Validate `current.updateID == openKey.expectedDataGeneration` -4. Commit if match, reject if mismatch (existing HDDS-10656 logic) +4. **Execution**: If validation passes, proceed with the operation (adding to OpenKeyTable). #### Error Mapping -| OM Error | S3 Status | S3 Error Code | Scenario | -|----------|-----------|---------------|----------| -| `KEY_ALREADY_EXISTS` | 412 | PreconditionFailed | If-None-Match failed | -| `KEY_NOT_FOUND` | 412 | PreconditionFailed | If-Match failed (key missing) | -| `ETAG_MISMATCH` | 412 | PreconditionFailed | If-Match failed (ETag mismatch) | -| `GENERATION_MISMATCH` | 412 | PreconditionFailed | If-Match failed (concurrent modification) | +| | | | | +|---|---|---|---| +|**OM Error**|**S3 Status**|**S3 Error Code**|**Scenario**| +|`KEY_ALREADY_EXISTS`|412|PreconditionFailed|If-None-Match failed| +|`KEY_NOT_FOUND`|412|PreconditionFailed|If-Match failed (key missing)| +|`ETAG_MISMATCH`|412|PreconditionFailed|If-Match failed (ETag mismatch)| +|`PRECONDITION_FAILED`|412|PreconditionFailed|If-Match failed (General/No ETag)| -## AWS S3 Conditional Read +## AWS S3 Conditional Read Implementation TODO -## AWS S3 Conditional Copy +## AWS S3 Conditional Copy Implementation TODO @@ -146,4 +175,6 @@ TODO - [AWS S3 Conditional Requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html) - [RFC 7232 - HTTP Conditional Requests](https://tools.ietf.org/html/rfc7232) - [HDDS-10656 - Atomic Rewrite Key](https://issues.apache.org/jira/browse/HDDS-10656) +- [HDDS-13963 - Atomic Create-If-Not-Exists](https://issues.apache.org/jira/browse/HDDS-13963) - [Leader Election with S3 Conditional Writes](https://www.morling.dev/blog/leader-election-with-s3-conditional-writes/) +- [An MVCC-like columnar table on S3 with constant-time deletes](https://simonwillison.net/2025/Oct/11/mvcc-s3/) From 7bfb33b8b51c5488bb68b95bfbe7c3334a7b9e42 Mon Sep 17 00:00:00 2001 From: peterxcli Date: Sun, 14 Dec 2025 15:11:16 +0800 Subject: [PATCH 4/7] Update S3 conditional requests documentation to reflect error code changes and ETag validation logic - Change error code from `KEY_ALREADY_EXISTS` to `KEY_GENERATION_MISMATCH` for concurrent key creation failures. - Modify ETag validation logic to allow operations to proceed when no ETag metadata is present, ensuring compatibility with mixed access patterns. - Update error mapping to include `ETAG_MISMATCH` for ETag comparison failures. - Add note regarding the upcoming addition of atomic create-if-not-exists capability linked to HDDS-13963. --- .../docs/content/design/s3-conditional-requests.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/hadoop-hdds/docs/content/design/s3-conditional-requests.md b/hadoop-hdds/docs/content/design/s3-conditional-requests.md index 1901649ce3ec..e5d5f6469680 100644 --- a/hadoop-hdds/docs/content/design/s3-conditional-requests.md +++ b/hadoop-hdds/docs/content/design/s3-conditional-requests.md @@ -119,13 +119,15 @@ public static final long EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS = -1L; ##### OM Commit Phase (Atomicity) 1. During the commit phase (or strict atomic create), the OM validates that the key still does not exist. -2. If a concurrent client created the key between the Create and Commit phases, the transaction fails with `KEY_ALREADY_EXISTS`. +2. If a concurrent client created the key between the Create and Commit phases, the transaction fails with `KET_GENERATION_MISMATCH`. ##### Race Condition Handling Using `OzoneConsts.EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS = -1` ensures atomicity. If a concurrent write (Client B) commits between Client A's Create and Commit, Client A's commit fails the `CREATE IF NOT EXISTS` validation check, preserving strict create-if-not-exists semantics. +> **Note**: This ability will be added along with [HDDS-13963](https://issues.apache.org/jira/browse/HDDS-13963) (Atomic Create-If-Not-Exists). + #### If-Match Implementation To optimize performance and reduce latency, we avoid a pre-flight check (GetS3KeyDetails) and instead validate the ETag during the OM Write operation. @@ -147,8 +149,8 @@ Validation is performed within the `validateAndUpdateCache` method to ensure ato 3. **Validation**: - **Key Not Found**: If the key does not exist, throw `KEY_NOT_FOUND` (maps to S3 412). - - **No ETag Metadata**: If the existing key (e.g., uploaded via OFS) does not have an ETag property, validation fails. We do **not** calculate ETag on the spot to avoid performance overhead on the applier thread. Throws `PRECONDITION_FAILED`. - - **ETag Mismatch**: Compare `existingKey.ETag` with `expectedETag`. If they do not match, throw `PRECONDITION_FAILED` (maps to S3 412). + - **No ETag Metadata**: If the existing key (e.g., uploaded via OFS) does not have an ETag property, skip ETag validation and allow the operation to proceed. This ensures compatibility with mixed access patterns (OFS and S3A) where S3 Conditional Writes are primarily intended for pure S3 use cases. We do **not** calculate ETag on the spot to avoid performance overhead on the applier thread. + - **ETag Mismatch**: Compare `existingKey.ETag` with `expectedETag`. If they do not match, throw `ETAG_MISMATCH` (maps to S3 412). 4. **Execution**: If validation passes, proceed with the operation (adding to OpenKeyTable). @@ -157,10 +159,9 @@ Validation is performed within the `validateAndUpdateCache` method to ensure ato | | | | | |---|---|---|---| |**OM Error**|**S3 Status**|**S3 Error Code**|**Scenario**| -|`KEY_ALREADY_EXISTS`|412|PreconditionFailed|If-None-Match failed| +|`KEY_GENERATION_MISMATCH`|412|PreconditionFailed|If-None-Match failed| |`KEY_NOT_FOUND`|412|PreconditionFailed|If-Match failed (key missing)| |`ETAG_MISMATCH`|412|PreconditionFailed|If-Match failed (ETag mismatch)| -|`PRECONDITION_FAILED`|412|PreconditionFailed|If-Match failed (General/No ETag)| ## AWS S3 Conditional Read Implementation From 1c1220963b32fd7cf6dec61ababc4cd2d65be937 Mon Sep 17 00:00:00 2001 From: peterxcli Date: Sun, 14 Dec 2025 15:12:20 +0800 Subject: [PATCH 5/7] Revert "WIP: Implement conditional write support in ObjectEndpoint for S3 API" This reverts commit dc046a18beaae8ae76cee12e14f6b6d66d16184c. --- .../ozone/s3/endpoint/ObjectEndpoint.java | 81 +--- .../ozone/client/ClientProtocolStub.java | 2 +- .../s3/endpoint/TestConditionalWrite.java | 384 ------------------ .../ozone/s3/endpoint/TestListParts.java | 6 +- .../endpoint/TestMultipartUploadComplete.java | 2 +- .../endpoint/TestMultipartUploadWithCopy.java | 10 +- .../ozone/s3/endpoint/TestObjectGet.java | 4 +- .../ozone/s3/endpoint/TestObjectPut.java | 70 ++-- .../s3/endpoint/TestObjectTaggingDelete.java | 2 +- .../s3/endpoint/TestObjectTaggingGet.java | 2 +- .../s3/endpoint/TestObjectTaggingPut.java | 14 +- .../ozone/s3/endpoint/TestPartUpload.java | 14 +- .../s3/endpoint/TestPartUploadWithStream.java | 8 +- .../s3/endpoint/TestPermissionCheck.java | 4 +- .../s3/endpoint/TestUploadWithStream.java | 4 +- .../s3/metrics/TestS3GatewayMetrics.java | 30 +- 16 files changed, 87 insertions(+), 550 deletions(-) delete mode 100644 hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestConditionalWrite.java diff --git a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java index 20b3df5c7853..b495ea346dc1 100644 --- a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java +++ b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java @@ -227,8 +227,6 @@ public Response put( @PathParam("bucket") String bucketName, @PathParam("path") String keyPath, @HeaderParam("Content-Length") long length, - @HeaderParam("If-None-Match") String ifNoneMatch, - @HeaderParam("If-Match") String ifMatch, @QueryParam("partNumber") int partNumber, @QueryParam("uploadId") @DefaultValue("") String uploadID, @QueryParam("tagging") String taggingMarker, @@ -313,49 +311,6 @@ public Response put( return Response.ok().status(HttpStatus.SC_OK).build(); } - // Check conditional write headers - // If-None-Match and If-Match are mutually exclusive - if (ifNoneMatch != null && ifMatch != null) { - throw newError(INVALID_ARGUMENT, keyPath); - } - - Long expectedGeneration = null; - - // Handle If-None-Match: * (create only if key doesn't exist) - if (ifNoneMatch != null) { - if (!"*".equals(ifNoneMatch.trim())) { - throw newError(INVALID_ARGUMENT, keyPath); - } - expectedGeneration = -1L; - } - - // Handle If-Match: (update only if ETag matches) - if (ifMatch != null) { - try { - // Look up existing key to get its ETag and generation - OzoneKeyDetails existingKey = getClientProtocol().getS3KeyDetails(bucketName, keyPath); - String existingETag = existingKey.getMetadata().get(OzoneConsts.ETAG); - - // Strip quotes from the provided ETag - String providedETag = stripQuotes(ifMatch.trim()); - - // Compare ETags - if (existingETag == null || !existingETag.equals(providedETag)) { - // ETag doesn't match - return 412 Precondition Failed - throw newError(S3ErrorTable.PRECOND_FAILED, keyPath); - } - - // ETag matches - use the key's generation for optimistic locking - expectedGeneration = existingKey.getGeneration(); - } catch (OMException ex) { - if (ex.getResult() == ResultCodes.KEY_NOT_FOUND) { - // Key doesn't exist - return 404 Not Found for If-Match - throw newError(S3ErrorTable.NO_SUCH_KEY, keyPath, ex); - } - throw ex; - } - } - // Normal put object S3ChunkInputStreamInfo chunkInputStreamInfo = getS3ChunkInputStreamInfo(body, length, amzDecodedLength, keyPath); @@ -376,21 +331,9 @@ public Response put( eTag = keyWriteResult.getKey(); putLength = keyWriteResult.getValue(); } else { - // Choose between rewriteKey (for If-Match and If-None-Match) or createKey (for normal put) - OzoneOutputStream output; - if (expectedGeneration != null) { - // If-Match: use rewriteKey with expectedDataGeneration for optimistic locking - output = getClientProtocol().rewriteKey( - volume.getName(), bucketName, keyPath, length, expectedGeneration, - replicationConfig, customMetadata); - } else { - // Normal create - output = getClientProtocol().createKey( - volume.getName(), bucketName, keyPath, length, replicationConfig, - customMetadata, tags); - } - - try { + try (OzoneOutputStream output = getClientProtocol().createKey( + volume.getName(), bucketName, keyPath, length, replicationConfig, + customMetadata, tags)) { long metadataLatencyNs = getMetrics().updatePutKeyMetadataStats(startNanos); perf.appendMetaLatencyNanos(metadataLatencyNs); @@ -400,8 +343,6 @@ public Response put( digestInputStream.getMessageDigest().digest()) .toLowerCase(); output.getMetadata().put(ETAG, eTag); - } finally { - output.close(); } } getMetrics().incPutKeySuccessLength(putLength); @@ -434,22 +375,6 @@ public Response put( throw newError(S3ErrorTable.QUOTA_EXCEEDED, keyPath, ex); } else if (ex.getResult() == ResultCodes.BUCKET_NOT_FOUND) { throw newError(S3ErrorTable.NO_SUCH_BUCKET, bucketName, ex); - } else if (ex.getResult() == ResultCodes.KEY_ALREADY_EXISTS) { - // If this is a conditional write failure, return 412 Precondition Failed - if (ifNoneMatch != null && "*".equals(ifNoneMatch.trim())) { - throw newError(S3ErrorTable.PRECOND_FAILED, keyPath, ex); - } - throw newError(S3ErrorTable.NO_OVERWRITE, keyPath, ex); - } else if (ex.getResult() == ResultCodes.KEY_NOT_FOUND) { - // For If-Match, if key doesn't exist during rewrite, return 404 - // This can happen if key was deleted between the initial check and the rewrite - if (ifMatch != null) { - throw newError(S3ErrorTable.NO_SUCH_KEY, keyPath, ex); - } - throw newError(S3ErrorTable.NO_SUCH_KEY, keyPath, ex); - } else if (ex.getResult() == ResultCodes.KEY_GENERATION_MISMATCH) { - // Generation mismatch during If-Match - return 412 Precondition Failed - throw newError(S3ErrorTable.PRECOND_FAILED, keyPath, ex); } else if (ex.getResult() == ResultCodes.FILE_ALREADY_EXISTS) { throw newError(S3ErrorTable.NO_OVERWRITE, keyPath, ex); } diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java index 8ccf6f9f3477..739babce1d06 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java @@ -717,7 +717,7 @@ public ListSnapshotResponse listSnapshot( String prevSnapshot, int maxListResult) throws IOException { return null; } - + @Override public void deleteSnapshot(String volumeName, String bucketName, String snapshotName) diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestConditionalWrite.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestConditionalWrite.java deleted file mode 100644 index f8898a15f622..000000000000 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestConditionalWrite.java +++ /dev/null @@ -1,384 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.ozone.s3.endpoint; - -import static java.nio.charset.StandardCharsets.UTF_8; -import static org.apache.hadoop.ozone.s3.util.S3Consts.X_AMZ_CONTENT_SHA256; -import static org.apache.hadoop.ozone.s3.util.S3Utils.stripQuotes; -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertNotNull; -import static org.junit.jupiter.api.Assertions.assertThrows; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.spy; -import static org.mockito.Mockito.when; - -import java.io.ByteArrayInputStream; -import java.io.IOException; -import java.io.InputStream; -import javax.ws.rs.core.HttpHeaders; -import javax.ws.rs.core.Response; -import org.apache.hadoop.hdds.conf.OzoneConfiguration; -import org.apache.hadoop.ozone.client.OzoneBucket; -import org.apache.hadoop.ozone.client.OzoneClient; -import org.apache.hadoop.ozone.client.OzoneClientStub; -import org.apache.hadoop.ozone.client.OzoneKeyDetails; -import org.apache.hadoop.ozone.s3.exception.OS3Exception; -import org.apache.hadoop.ozone.s3.exception.S3ErrorTable; -import org.apache.http.HttpStatus; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; - -/** - * Test conditional writes (If-Match, If-None-Match) for PutObject. - */ -class TestConditionalWrite { - private static final String BUCKET_NAME = "test-bucket"; - private static final String KEY_NAME = "test-key"; - private static final String CONTENT = "test content"; - - private OzoneClient clientStub; - private ObjectEndpoint objectEndpoint; - private HttpHeaders headers; - private OzoneBucket bucket; - - @BeforeEach - void setup() throws IOException { - OzoneConfiguration config = new OzoneConfiguration(); - - // Create client stub and object store stub - clientStub = new OzoneClientStub(); - - // Create bucket - clientStub.getObjectStore().createS3Bucket(BUCKET_NAME); - bucket = clientStub.getObjectStore().getS3Bucket(BUCKET_NAME); - - headers = mock(HttpHeaders.class); - when(headers.getHeaderString(X_AMZ_CONTENT_SHA256)).thenReturn("mockSignature"); - - // Create ObjectEndpoint and set client to OzoneClientStub - objectEndpoint = EndpointBuilder.newObjectEndpointBuilder() - .setClient(clientStub) - .setConfig(config) - .setHeaders(headers) - .build(); - - objectEndpoint = spy(objectEndpoint); - } - - /** - * Test If-None-Match: * succeeds when key doesn't exist. - */ - @Test - void testIfNoneMatchSuccessWhenKeyNotExists() throws Exception { - InputStream inputStream = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); - - Response response = objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - CONTENT.length(), - "*", // If-None-Match: * - null, // If-Match - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream - ); - - assertEquals(HttpStatus.SC_OK, response.getStatus()); - assertNotNull(response.getHeaderString("ETag")); - - // Verify key was created - OzoneKeyDetails keyDetails = bucket.getKey(KEY_NAME); - assertNotNull(keyDetails); - } - - /** - * Test If-None-Match: * fails when key already exists. - */ - @Test - void testIfNoneMatchFailsWhenKeyExists() throws Exception { - // First, create the key normally - InputStream inputStream1 = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); - Response response1 = objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - CONTENT.length(), - null, // If-None-Match - null, // If-Match - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream1 - ); - assertEquals(HttpStatus.SC_OK, response1.getStatus()); - - // Second, try to create the same key with If-None-Match: * - InputStream inputStream2 = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); - OS3Exception exception = assertThrows(OS3Exception.class, () -> { - objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - CONTENT.length(), - "*", // If-None-Match: * - null, // If-Match - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream2 - ); - }); - - assertEquals(HttpStatus.SC_PRECONDITION_FAILED, exception.getHttpCode()); - assertEquals(S3ErrorTable.PRECOND_FAILED.getCode(), exception.getCode()); - } - - /** - * Test If-Match succeeds when ETag matches. - */ - @Test - void testIfMatchSuccessWhenETagMatches() throws Exception { - // First, create the key normally - InputStream inputStream1 = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); - Response response1 = objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - CONTENT.length(), - null, // If-None-Match - null, // If-Match - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream1 - ); - assertEquals(HttpStatus.SC_OK, response1.getStatus()); - String etag = stripQuotes(response1.getHeaderString("ETag")); - - // Second, update the key with If-Match using the correct ETag - String newContent = "updated content"; - InputStream inputStream2 = new ByteArrayInputStream(newContent.getBytes(UTF_8)); - Response response2 = objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - newContent.length(), - null, // If-None-Match - "\"" + etag + "\"", // If-Match with quoted ETag - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream2 - ); - - assertEquals(HttpStatus.SC_OK, response2.getStatus()); - assertNotNull(response2.getHeaderString("ETag")); - } - - /** - * Test If-Match fails when key doesn't exist. - */ - @Test - void testIfMatchFailsWhenKeyNotExists() throws Exception { - String fakeEtag = "abc123"; - InputStream inputStream = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); - - OS3Exception exception = assertThrows(OS3Exception.class, () -> { - objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - CONTENT.length(), - null, // If-None-Match - "\"" + fakeEtag + "\"", // If-Match - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream - ); - }); - - assertEquals(HttpStatus.SC_NOT_FOUND, exception.getHttpCode()); - assertEquals(S3ErrorTable.NO_SUCH_KEY.getCode(), exception.getCode()); - } - - /** - * Test If-Match fails when ETag doesn't match. - */ - @Test - void testIfMatchFailsWhenETagMismatch() throws Exception { - // First, create the key normally - InputStream inputStream1 = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); - Response response1 = objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - CONTENT.length(), - null, // If-None-Match - null, // If-Match - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream1 - ); - assertEquals(HttpStatus.SC_OK, response1.getStatus()); - - // Second, try to update with wrong ETag - String wrongEtag = "wrongetag123"; - String newContent = "updated content"; - InputStream inputStream2 = new ByteArrayInputStream(newContent.getBytes(UTF_8)); - - OS3Exception exception = assertThrows(OS3Exception.class, () -> { - objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - newContent.length(), - null, // If-None-Match - "\"" + wrongEtag + "\"", // If-Match with wrong ETag - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream2 - ); - }); - - assertEquals(HttpStatus.SC_PRECONDITION_FAILED, exception.getHttpCode()); - assertEquals(S3ErrorTable.PRECOND_FAILED.getCode(), exception.getCode()); - } - - /** - * Test that If-Match and If-None-Match are mutually exclusive. - */ - @Test - void testIfMatchAndIfNoneMatchMutuallyExclusive() throws Exception { - InputStream inputStream = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); - - OS3Exception exception = assertThrows(OS3Exception.class, () -> { - objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - CONTENT.length(), - "*", // If-None-Match - "\"abc\"", // If-Match - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream - ); - }); - - assertEquals(HttpStatus.SC_BAD_REQUEST, exception.getHttpCode()); - assertEquals(S3ErrorTable.INVALID_ARGUMENT.getCode(), exception.getCode()); - } - - /** - * Test If-None-Match with invalid value (must be "*"). - */ - @Test - void testIfNoneMatchInvalidValue() throws Exception { - InputStream inputStream = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); - - OS3Exception exception = assertThrows(OS3Exception.class, () -> { - objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - CONTENT.length(), - "\"etag\"", // If-None-Match with invalid value (must be "*") - null, // If-Match - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream - ); - }); - - assertEquals(HttpStatus.SC_BAD_REQUEST, exception.getHttpCode()); - assertEquals(S3ErrorTable.INVALID_ARGUMENT.getCode(), exception.getCode()); - } - - /** - * Test generation mismatch detection - simulates concurrent modification. - * This test verifies that when a key is modified between the If-Match check - * and the actual rewrite, it returns 412 Precondition Failed. - */ - @Test - void testIfMatchGenerationMismatch() throws Exception { - // First, create the key normally - InputStream inputStream1 = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); - Response response1 = objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - CONTENT.length(), - null, // If-None-Match - null, // If-Match - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream1 - ); - assertEquals(HttpStatus.SC_OK, response1.getStatus()); - String etag = stripQuotes(response1.getHeaderString("ETag")); - - // Second, update the key normally (simulating concurrent modification) - String intermediateContent = "intermediate update"; - InputStream inputStream2 = new ByteArrayInputStream(intermediateContent.getBytes(UTF_8)); - Response response2 = objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - intermediateContent.length(), - null, // If-None-Match - null, // If-Match - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream2 - ); - assertEquals(HttpStatus.SC_OK, response2.getStatus()); - - // Third, try to update using the old ETag (should fail with generation mismatch) - String finalContent = "final update"; - InputStream inputStream3 = new ByteArrayInputStream(finalContent.getBytes(UTF_8)); - - OS3Exception exception = assertThrows(OS3Exception.class, () -> { - objectEndpoint.put( - BUCKET_NAME, - KEY_NAME, - finalContent.length(), - null, // If-None-Match - "\"" + etag + "\"", // If-Match with old ETag - 0, // partNumber - "", // uploadID - null, // taggingMarker - null, // aclMarker - inputStream3 - ); - }); - - // Should return 412 Precondition Failed (not 404) - assertEquals(HttpStatus.SC_PRECONDITION_FAILED, exception.getHttpCode()); - assertEquals(S3ErrorTable.PRECOND_FAILED.getCode(), exception.getCode()); - } -} - diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestListParts.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestListParts.java index 0166cda24473..30be715b5305 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestListParts.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestListParts.java @@ -76,17 +76,17 @@ public void setUp() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), null, null, 1, uploadID, null, null, body); + content.length(), 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), null, null, 2, uploadID, null, null, body); + content.length(), 2, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), null, null, 3, uploadID, null, null, body); + content.length(), 3, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); } diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadComplete.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadComplete.java index abedf437cf79..fde336f48079 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadComplete.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadComplete.java @@ -107,7 +107,7 @@ private Part uploadPart(String key, String uploadID, int partNumber, String ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); Response response = rest.put(OzoneConsts.S3_BUCKET, key, content.length(), - null, null, partNumber, uploadID, null, null, body); + partNumber, uploadID, null, null, body); assertEquals(200, response.getStatus()); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); Part part = new Part(); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadWithCopy.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadWithCopy.java index 085591e4c241..fd83523214ca 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadWithCopy.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadWithCopy.java @@ -277,7 +277,7 @@ public enum CopyIfTimestampTestCase { MODIFIED_SINCE_FUTURE_TS_UNMODIFIED_SINCE_UNPARSABLE_TS( futureTimeStr, UNPARSABLE_TIME_STR, null); - + private final String modifiedTimestamp; private final String unmodifiedTimestamp; private final String errorCode; @@ -336,7 +336,7 @@ private Part uploadPart(String key, String uploadID, int partNumber, String ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); Response response = REST.put(OzoneConsts.S3_BUCKET, key, content.length(), - null, null, partNumber, uploadID, null, null, body); + partNumber, uploadID, null, null, body); assertEquals(200, response.getStatus()); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); Part part = new Part(); @@ -380,8 +380,8 @@ private Part uploadPartWithCopy(String key, String uploadID, int partNumber, setHeaders(additionalHeaders); ByteArrayInputStream body = new ByteArrayInputStream("".getBytes(UTF_8)); - Response response = REST.put(OzoneConsts.S3_BUCKET, key, 0, null, null, - partNumber, uploadID, null, null, body); + Response response = REST.put(OzoneConsts.S3_BUCKET, key, 0, partNumber, + uploadID, null, null, body); assertEquals(200, response.getStatus()); CopyPartResult result = (CopyPartResult) response.getEntity(); @@ -408,7 +408,7 @@ public void testUploadWithRangeCopyContentLength() OzoneConsts.S3_BUCKET + "/" + EXISTING_KEY); additionalHeaders.put(COPY_SOURCE_HEADER_RANGE, "bytes=0-3"); setHeaders(additionalHeaders); - REST.put(OzoneConsts.S3_BUCKET, KEY, 0, null, null, 1, uploadID, null, null, body); + REST.put(OzoneConsts.S3_BUCKET, KEY, 0, 1, uploadID, null, null, body); OzoneMultipartUploadPartListParts parts = CLIENT.getObjectStore().getS3Bucket(OzoneConsts.S3_BUCKET) .listParts(KEY, uploadID, 0, 100); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectGet.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectGet.java index b8d73b85c79d..3e772f8b8bf7 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectGet.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectGet.java @@ -93,11 +93,11 @@ public void init() throws OS3Exception, IOException { ByteArrayInputStream body = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); rest.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - null, null, 1, null, null, null, body); + 1, null, null, null, body); // Create a key with object tags when(headers.getHeaderString(TAG_HEADER)).thenReturn("tag1=value1&tag2=value2"); rest.put(BUCKET_NAME, KEY_WITH_TAG, CONTENT.length(), - null, null, 1, null, null, null, body); + 1, null, null, null, body); context = mock(ContainerRequestContext.class); when(context.getUriInfo()).thenReturn(mock(UriInfo.class)); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectPut.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectPut.java index 11722fb46893..e5c34fb4e465 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectPut.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectPut.java @@ -158,7 +158,7 @@ void testPutObject(int length, ReplicationConfig replication) throws IOException bucket.setReplicationConfig(replication); //WHEN - Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, length, null, null, 1, null, null, null, body); + Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, length, 1, null, null, null, body); //THEN assertEquals(200, response.getStatus()); @@ -185,7 +185,7 @@ void testPutObjectContentLength() throws IOException, OS3Exception { new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); long dataSize = CONTENT.length(); - objectEndpoint.put(BUCKET_NAME, KEY_NAME, dataSize, null, null, 0, null, null, null, body); + objectEndpoint.put(BUCKET_NAME, KEY_NAME, dataSize, 0, null, null, null, body); assertEquals(dataSize, getKeyDataSize()); } @@ -202,7 +202,7 @@ void testPutObjectContentLengthForStreaming() when(headers.getHeaderString(DECODED_CONTENT_LENGTH_HEADER)) .thenReturn("15"); - objectEndpoint.put(BUCKET_NAME, KEY_NAME, chunkedContent.length(), null, null, 0, null, null, + objectEndpoint.put(BUCKET_NAME, KEY_NAME, chunkedContent.length(), 0, null, null, null, new ByteArrayInputStream(chunkedContent.getBytes(UTF_8))); assertEquals(15, getKeyDataSize()); } @@ -218,7 +218,7 @@ public void testPutObjectWithTags() throws IOException, OS3Exception { objectEndpoint.setHeaders(headersWithTags); Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - null, null, 1, null, null, null, body); + 1, null, null, null, body); assertEquals(200, response.getStatus()); @@ -242,7 +242,7 @@ public void testPutObjectWithOnlyTagKey() throws Exception { try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - null, null, 1, null, null, null, body); + 1, null, null, null, body); fail("request with invalid query param should fail"); } catch (OS3Exception ex) { assertEquals(INVALID_TAG.getCode(), ex.getCode()); @@ -261,7 +261,7 @@ public void testPutObjectWithDuplicateTagKey() throws Exception { objectEndpoint.setHeaders(headersWithDuplicateTagKey); try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - null, null, 1, null, null, null, body); + 1, null, null, null, body); fail("request with duplicate tag key should fail"); } catch (OS3Exception ex) { assertEquals(INVALID_TAG.getCode(), ex.getCode()); @@ -281,7 +281,7 @@ public void testPutObjectWithLongTagKey() throws Exception { objectEndpoint.setHeaders(headersWithLongTagKey); try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - null, null, 1, null, null, null, body); + 1, null, null, null, body); fail("request with tag key exceeding the length limit should fail"); } catch (OS3Exception ex) { assertEquals(INVALID_TAG.getCode(), ex.getCode()); @@ -301,7 +301,7 @@ public void testPutObjectWithLongTagValue() throws Exception { when(headersWithLongTagValue.getHeaderString(TAG_HEADER)).thenReturn("tag1=" + longTagValue); try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - null, null, 1, null, null, null, body); + 1, null, null, null, body); fail("request with tag value exceeding the length limit should fail"); } catch (OS3Exception ex) { assertEquals(INVALID_TAG.getCode(), ex.getCode()); @@ -327,7 +327,7 @@ public void testPutObjectWithTooManyTags() throws Exception { objectEndpoint.setHeaders(headersWithTooManyTags); try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT.length(), - null, null, 1, null, null, null, body); + 1, null, null, null, body); fail("request with number of tags exceeding limit should fail"); } catch (OS3Exception ex) { assertEquals(INVALID_TAG.getCode(), ex.getCode()); @@ -356,7 +356,7 @@ void testPutObjectWithSignedChunks() throws IOException, OS3Exception { //WHEN Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, - chunkedContent.length(), null, null, 1, null, null, null, + chunkedContent.length(), 1, null, null, null, new ByteArrayInputStream(chunkedContent.getBytes(UTF_8))); //THEN @@ -386,7 +386,7 @@ public void testPutObjectMessageDigestResetDuringException() throws OS3Exception new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); try { objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT - .length(), null, null, 1, null, null, null, body); + .length(), 1, null, null, null, body); fail("Should throw IOException"); } catch (IOException ignored) { // Verify that the message digest is reset so that the instance can be reused for the @@ -411,7 +411,7 @@ void testCopyObject() throws IOException, OS3Exception { when(headers.getHeaderString(CUSTOM_METADATA_COPY_DIRECTIVE_HEADER)).thenReturn("COPY"); Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, - CONTENT.length(), null, null, 1, null, null, null, body); + CONTENT.length(), 1, null, null, null, body); OzoneInputStream ozoneInputStream = clientStub.getObjectStore() .getS3Bucket(BUCKET_NAME) @@ -436,7 +436,7 @@ void testCopyObject() throws IOException, OS3Exception { when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( BUCKET_NAME + "/" + urlEncode(KEY_NAME)); - response = objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), null, null, 1, + response = objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), 1, null, null, null, body); // Check destination key and response @@ -466,7 +466,7 @@ void testCopyObject() throws IOException, OS3Exception { metadataHeaders.remove(CUSTOM_METADATA_HEADER_PREFIX + "custom-key-1"); metadataHeaders.remove(CUSTOM_METADATA_HEADER_PREFIX + "custom-key-2"); - response = objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), null, null, 1, + response = objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), 1, null, null, null, body); ozoneInputStream = clientStub.getObjectStore().getS3Bucket(DEST_BUCKET_NAME) @@ -494,7 +494,7 @@ void testCopyObject() throws IOException, OS3Exception { // wrong copy metadata directive when(headers.getHeaderString(CUSTOM_METADATA_COPY_DIRECTIVE_HEADER)).thenReturn("INVALID"); OS3Exception e = assertThrows(OS3Exception.class, () -> objectEndpoint.put( - DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), null, null, 1, null, null, null, body), + DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), 1, null, null, null, body), "test copy object failed"); assertThat(e.getHttpCode()).isEqualTo(400); assertThat(e.getCode()).isEqualTo("InvalidArgument"); @@ -504,7 +504,7 @@ void testCopyObject() throws IOException, OS3Exception { // source and dest same e = assertThrows(OS3Exception.class, () -> objectEndpoint.put( - BUCKET_NAME, KEY_NAME, CONTENT.length(), null, null, 1, null, null, null, body), + BUCKET_NAME, KEY_NAME, CONTENT.length(), 1, null, null, null, body), "test copy object failed"); assertThat(e.getErrorMessage()).contains("This copy request is illegal"); @@ -512,28 +512,28 @@ void testCopyObject() throws IOException, OS3Exception { when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( NO_SUCH_BUCKET + "/" + urlEncode(KEY_NAME)); e = assertThrows(OS3Exception.class, () -> objectEndpoint.put(DEST_BUCKET_NAME, - DEST_KEY, CONTENT.length(), null, null, 1, null, null, null, body), "test copy object failed"); + DEST_KEY, CONTENT.length(), 1, null, null, null, body), "test copy object failed"); assertThat(e.getCode()).contains("NoSuchBucket"); // dest bucket not found when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( BUCKET_NAME + "/" + urlEncode(KEY_NAME)); e = assertThrows(OS3Exception.class, () -> objectEndpoint.put(NO_SUCH_BUCKET, - DEST_KEY, CONTENT.length(), null, null, 1, null, null, null, body), "test copy object failed"); + DEST_KEY, CONTENT.length(), 1, null, null, null, body), "test copy object failed"); assertThat(e.getCode()).contains("NoSuchBucket"); //Both source and dest bucket not found when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( NO_SUCH_BUCKET + "/" + urlEncode(KEY_NAME)); e = assertThrows(OS3Exception.class, () -> objectEndpoint.put(NO_SUCH_BUCKET, - DEST_KEY, CONTENT.length(), null, null, 1, null, null, null, body), "test copy object failed"); + DEST_KEY, CONTENT.length(), 1, null, null, null, body), "test copy object failed"); assertThat(e.getCode()).contains("NoSuchBucket"); // source key not found when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( BUCKET_NAME + "/" + urlEncode(NO_SUCH_BUCKET)); e = assertThrows(OS3Exception.class, () -> objectEndpoint.put( - "nonexistent", KEY_NAME, CONTENT.length(), null, null, 1, null, null, null, body), + "nonexistent", KEY_NAME, CONTENT.length(), 1, null, null, null, body), "test copy object failed"); assertThat(e.getCode()).contains("NoSuchBucket"); } @@ -545,7 +545,7 @@ public void testCopyObjectMessageDigestResetDuringException() throws IOException new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); Response response = objectEndpoint.put(BUCKET_NAME, KEY_NAME, - CONTENT.length(), null, null, 1, null, null, null, body); + CONTENT.length(), 1, null, null, null, body); OzoneInputStream ozoneInputStream = clientStub.getObjectStore() .getS3Bucket(BUCKET_NAME) @@ -572,7 +572,7 @@ public void testCopyObjectMessageDigestResetDuringException() throws IOException BUCKET_NAME + "/" + urlEncode(KEY_NAME)); try { - objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), null, null, 1, + objectEndpoint.put(DEST_BUCKET_NAME, DEST_KEY, CONTENT.length(), 1, null, null, null, body); fail("Should throw IOException"); } catch (IOException ignored) { @@ -596,7 +596,7 @@ public void testCopyObjectWithTags() throws IOException, OS3Exception { String sourceKeyName = "sourceKey"; Response putResponse = objectEndpoint.put(BUCKET_NAME, sourceKeyName, - CONTENT.length(), null, null, 1, null, null, null, body); + CONTENT.length(), 1, null, null, null, body); OzoneKeyDetails keyDetails = clientStub.getObjectStore().getS3Bucket(BUCKET_NAME).getKey(sourceKeyName); @@ -614,8 +614,7 @@ public void testCopyObjectWithTags() throws IOException, OS3Exception { BUCKET_NAME + "/" + urlEncode(sourceKeyName)); objectEndpoint.setHeaders(headersForCopy); - Response copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), null, null, 1, null, null, - null, body); + Response copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), 1, null, null, null, body); OzoneKeyDetails destKeyDetails = clientStub.getObjectStore() .getS3Bucket(DEST_BUCKET_NAME).getKey(destKey); @@ -634,8 +633,7 @@ public void testCopyObjectWithTags() throws IOException, OS3Exception { // With x-amz-tagging-directive = COPY with a different x-amz-tagging when(headersForCopy.getHeaderString(TAG_HEADER)).thenReturn("tag3=value3"); - copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), null, null, 1, null, null, null, - body); + copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), 1, null, null, null, body); assertEquals(200, copyResponse.getStatus()); destKeyDetails = clientStub.getObjectStore() @@ -650,8 +648,7 @@ public void testCopyObjectWithTags() throws IOException, OS3Exception { // Copy object with x-amz-tagging-directive = REPLACE when(headersForCopy.getHeaderString(TAG_DIRECTIVE_HEADER)).thenReturn("REPLACE"); - copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), null, null, 1, null, null, null, - body); + copyResponse = objectEndpoint.put(DEST_BUCKET_NAME, destKey, CONTENT.length(), 1, null, null, null, body); assertEquals(200, copyResponse.getStatus()); destKeyDetails = clientStub.getObjectStore() @@ -673,7 +670,7 @@ public void testCopyObjectWithInvalidTagCopyDirective() throws Exception { HttpHeaders headersForCopy = Mockito.mock(HttpHeaders.class); when(headersForCopy.getHeaderString(TAG_DIRECTIVE_HEADER)).thenReturn("INVALID"); try { - objectEndpoint.put(DEST_BUCKET_NAME, "somekey", CONTENT.length(), null, null, 1, null, null, null, body); + objectEndpoint.put(DEST_BUCKET_NAME, "somekey", CONTENT.length(), 1, null, null, null, body); } catch (OS3Exception ex) { assertEquals(INVALID_ARGUMENT.getCode(), ex.getCode()); assertThat(ex.getErrorMessage()).contains("The tagging copy directive specified is invalid"); @@ -688,7 +685,7 @@ void testInvalidStorageType() { when(headers.getHeaderString(STORAGE_CLASS_HEADER)).thenReturn("random"); OS3Exception e = assertThrows(OS3Exception.class, () -> objectEndpoint.put( - BUCKET_NAME, KEY_NAME, CONTENT.length(), null, null, 1, null, null, null, body)); + BUCKET_NAME, KEY_NAME, CONTENT.length(), 1, null, null, null, body)); assertEquals(S3ErrorTable.INVALID_STORAGE_CLASS.getErrorMessage(), e.getErrorMessage()); assertEquals("random", e.getResource()); @@ -701,7 +698,7 @@ void testEmptyStorageType() throws IOException, OS3Exception { when(headers.getHeaderString(STORAGE_CLASS_HEADER)).thenReturn(""); objectEndpoint.put(BUCKET_NAME, KEY_NAME, CONTENT - .length(), null, null, 1, null, null, null, body); + .length(), 1, null, null, null, body); OzoneKeyDetails key = clientStub.getObjectStore().getS3Bucket(BUCKET_NAME) .getKey(KEY_NAME); @@ -720,7 +717,7 @@ void testDirectoryCreation() throws IOException, // WHEN try (Response response = objectEndpoint.put(fsoBucket.getName(), path, - 0L, null, null, 0, "", null, null, null)) { + 0L, 0, "", null, null, null)) { assertEquals(HttpStatus.SC_OK, response.getStatus()); } @@ -735,12 +732,12 @@ void testDirectoryCreationOverFile() throws IOException, OS3Exception { final String path = "key"; final ByteArrayInputStream body = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); - objectEndpoint.put(FSO_BUCKET_NAME, path, CONTENT.length(), null, null, 0, "", null, null, body); + objectEndpoint.put(FSO_BUCKET_NAME, path, CONTENT.length(), 0, "", null, null, body); // WHEN final OS3Exception exception = assertThrows(OS3Exception.class, () -> objectEndpoint - .put(FSO_BUCKET_NAME, path + "/", 0, null, null, 0, "", null, null, null) + .put(FSO_BUCKET_NAME, path + "/", 0, 0, "", null, null, null) .close()); // THEN @@ -756,8 +753,7 @@ public void testPutEmptyObject() throws IOException, OS3Exception { ByteArrayInputStream body = new ByteArrayInputStream(emptyString.getBytes(UTF_8)); objectEndpoint.setHeaders(headersWithTags); - Response putResponse = objectEndpoint.put(BUCKET_NAME, KEY_NAME, emptyString.length(), null, null, 1, null, null, - null, body); + Response putResponse = objectEndpoint.put(BUCKET_NAME, KEY_NAME, emptyString.length(), 1, null, null, null, body); assertEquals(200, putResponse.getStatus()); OzoneKeyDetails keyDetails = clientStub.getObjectStore().getS3Bucket(BUCKET_NAME).getKey(KEY_NAME); assertEquals(0, keyDetails.getDataSize()); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingDelete.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingDelete.java index 8c249566b290..81a260ff4f21 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingDelete.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingDelete.java @@ -84,7 +84,7 @@ public void init() throws OS3Exception, IOException { Mockito.when(headers.getHeaderString(X_AMZ_CONTENT_SHA256)) .thenReturn("mockSignature"); rest.put(BUCKET_NAME, KEY_WITH_TAG, CONTENT.length(), - null, null, 1, null, null, null, body); + 1, null, null, null, body); ContainerRequestContext context = Mockito.mock(ContainerRequestContext.class); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingGet.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingGet.java index 6e3d1823e4e2..c4eb4c25ff87 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingGet.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingGet.java @@ -75,7 +75,7 @@ public void init() throws OS3Exception, IOException { // Create a key with object tags Mockito.when(headers.getHeaderString(TAG_HEADER)).thenReturn("tag1=value1&tag2=value2"); rest.put(BUCKET_NAME, KEY_WITH_TAG, CONTENT.length(), - null, null, 1, null, null, null, body); + 1, null, null, null, body); ContainerRequestContext context = Mockito.mock(ContainerRequestContext.class); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingPut.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingPut.java index 5f56aff66a70..02b71e8772c4 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingPut.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectTaggingPut.java @@ -89,13 +89,13 @@ void setup() throws IOException, OS3Exception { ByteArrayInputStream body = new ByteArrayInputStream("".getBytes(UTF_8)); - objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, null, null, 1, null, null, null, body); + objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, 1, null, null, null, body); } @Test public void testPutObjectTaggingWithEmptyBody() throws Exception { try { - objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, null, null, 1, null, "", null, + objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, 1, null, "", null, null); fail(); } catch (OS3Exception ex) { @@ -106,7 +106,7 @@ public void testPutObjectTaggingWithEmptyBody() throws Exception { @Test public void testPutValidObjectTagging() throws Exception { - assertEquals(HTTP_OK, objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, null, null, 1, null, + assertEquals(HTTP_OK, objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, 1, null, "", null, twoTags()).getStatus()); OzoneKeyDetails keyDetails = clientStub.getObjectStore().getS3Bucket(BUCKET_NAME).getKey(KEY_NAME); @@ -128,7 +128,7 @@ public void testPutInvalidObjectTagging() throws Exception { private void testInvalidObjectTagging(Supplier inputStream, int expectedHttpCode, String expectedErrorCode) throws Exception { try { - objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, null, null, 1, null, "", null, + objectEndpoint.put(BUCKET_NAME, KEY_NAME, 0, 1, null, "", null, inputStream.get()); fail("Expected an OS3Exception to be thrown"); } catch (OS3Exception ex) { @@ -140,7 +140,7 @@ private void testInvalidObjectTagging(Supplier inputStream, @Test public void testPutObjectTaggingNoKeyFound() throws Exception { try { - objectEndpoint.put(BUCKET_NAME, "nonexistent", 0, null, null, 1, + objectEndpoint.put(BUCKET_NAME, "nonexistent", 0, 1, null, "", null, twoTags()); fail("Expected an OS3Exception to be thrown"); } catch (OS3Exception ex) { @@ -152,7 +152,7 @@ public void testPutObjectTaggingNoKeyFound() throws Exception { @Test public void testPutObjectTaggingNoBucketFound() throws Exception { try { - objectEndpoint.put("nonexistent", "nonexistent", 0, null, null, 1, + objectEndpoint.put("nonexistent", "nonexistent", 0, 1, null, "", null, twoTags()); fail("Expected an OS3Exception to be thrown"); } catch (OS3Exception ex) { @@ -184,7 +184,7 @@ public void testPutObjectTaggingNotImplemented() throws Exception { ResultCodes.NOT_SUPPORTED_OPERATION)).when(mockBucket).putObjectTagging("dir/", twoTagsMap); try { - endpoint.put("fsoBucket", "dir/", 0, null, null, 1, null, "", + endpoint.put("fsoBucket", "dir/", 0, 1, null, "", null, twoTags()); fail("Expected an OS3Exception to be thrown"); } catch (OS3Exception ex) { diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUpload.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUpload.java index dd206f06bae2..4981069528a8 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUpload.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUpload.java @@ -97,7 +97,7 @@ public void testPartUpload() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), null, null, 1, uploadID, null, null, body); + content.length(), 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); @@ -119,7 +119,7 @@ public void testPartUploadWithOverride() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), null, null, 1, uploadID, null, null, body); + content.length(), 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); @@ -128,7 +128,7 @@ public void testPartUploadWithOverride() throws Exception { // Upload part again with same part Number, the ETag should be changed. content = "Multipart Upload Changed"; response = rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), null, null, 1, uploadID, null, null, body); + content.length(), 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); assertNotEquals(eTag, response.getHeaderString(OzoneConsts.ETAG)); @@ -140,7 +140,7 @@ public void testPartUploadWithIncorrectUploadID() throws Exception { String content = "Multipart Upload With Incorrect uploadID"; ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); - rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, content.length(), null, null, 1, + rest.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, content.length(), 1, "random", null, null, body); }); assertEquals("NoSuchUpload", ex.getCode()); @@ -176,7 +176,7 @@ public void testPartUploadStreamContentLength() String uploadID = multipartUploadInitiateResponse.getUploadID(); long contentLength = chunkedContent.length(); - objectEndpoint.put(OzoneConsts.S3_BUCKET, keyName, contentLength, null, null, 1, + objectEndpoint.put(OzoneConsts.S3_BUCKET, keyName, contentLength, 1, uploadID, null, null, new ByteArrayInputStream(chunkedContent.getBytes(UTF_8))); assertContentLength(uploadID, keyName, 15); } @@ -200,7 +200,7 @@ public void testPartUploadContentLength() throws IOException, OS3Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); rest.put(OzoneConsts.S3_BUCKET, keyName, - contentLength, null, null, 1, uploadID, null, null, body); + contentLength, 1, uploadID, null, null, body); assertContentLength(uploadID, keyName, content.length()); } @@ -245,7 +245,7 @@ public void testPartUploadMessageDigestResetDuringException() throws IOException new ByteArrayInputStream(content.getBytes(UTF_8)); try { objectEndpoint.put(OzoneConsts.S3_BUCKET, OzoneConsts.KEY, - content.length(), null, null, 1, uploadID, null, null, body); + content.length(), 1, uploadID, null, null, body); fail("Should throw IOException"); } catch (IOException ignored) { // Verify that the message digest is reset so that the instance can be reused for the diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUploadWithStream.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUploadWithStream.java index ccdad9ada3b5..4b2d8a49efb9 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUploadWithStream.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUploadWithStream.java @@ -96,7 +96,7 @@ public void testPartUpload() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); response = rest.put(S3BUCKET, S3KEY, - content.length(), null, null, 1, uploadID, null, null, body); + content.length(), 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); @@ -117,7 +117,7 @@ public void testPartUploadWithOverride() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); response = rest.put(S3BUCKET, S3KEY, - content.length(), null, null, 1, uploadID, null, null, body); + content.length(), 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); @@ -126,7 +126,7 @@ public void testPartUploadWithOverride() throws Exception { // Upload part again with same part Number, the ETag should be changed. content = "Multipart Upload Changed"; response = rest.put(S3BUCKET, S3KEY, - content.length(), null, null, 1, uploadID, null, null, body); + content.length(), 1, uploadID, null, null, body); assertNotNull(response.getHeaderString(OzoneConsts.ETAG)); assertNotEquals(eTag, response.getHeaderString(OzoneConsts.ETAG)); @@ -138,7 +138,7 @@ public void testPartUploadWithIncorrectUploadID() throws Exception { String content = "Multipart Upload With Incorrect uploadID"; ByteArrayInputStream body = new ByteArrayInputStream(content.getBytes(UTF_8)); - rest.put(S3BUCKET, S3KEY, content.length(), null, null, 1, + rest.put(S3BUCKET, S3KEY, content.length(), 1, "random", null, null, body); }); assertEquals("NoSuchUpload", ex.getCode()); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPermissionCheck.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPermissionCheck.java index f4768552d947..81f6853bf73f 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPermissionCheck.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPermissionCheck.java @@ -277,7 +277,7 @@ public void testPutKey() throws IOException { .build(); OS3Exception e = assertThrows(OS3Exception.class, () -> objectEndpoint.put( - "bucketName", "keyPath", 1024, null, null, 0, null, null, null, + "bucketName", "keyPath", 1024, 0, null, null, null, new ByteArrayInputStream(new byte[]{}))); assertEquals(HTTP_FORBIDDEN, e.getHttpCode()); } @@ -340,7 +340,7 @@ public void testObjectTagging() throws Exception { InputStream tagInput = new ByteArrayInputStream(xml.getBytes(UTF_8)); OS3Exception e = assertThrows(OS3Exception.class, () -> - objectEndpoint.put("bucketName", "keyPath", 0, null, null, 1, + objectEndpoint.put("bucketName", "keyPath", 0, 1, null, "", null, tagInput)); assertEquals(HTTP_FORBIDDEN, e.getHttpCode()); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestUploadWithStream.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestUploadWithStream.java index adefc07b2e63..7ed2c488c444 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestUploadWithStream.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestUploadWithStream.java @@ -103,7 +103,7 @@ public void testUpload() throws Exception { byte[] keyContent = S3_COPY_EXISTING_KEY_CONTENT.getBytes(UTF_8); ByteArrayInputStream body = new ByteArrayInputStream(keyContent); - Response response = rest.put(S3BUCKET, S3KEY, 0, null, null, 0, null, null, null, body); + Response response = rest.put(S3BUCKET, S3KEY, 0, 0, null, null, null, body); assertEquals(200, response.getStatus()); } @@ -137,7 +137,7 @@ public void testUploadWithCopy() throws Exception { .forEach((k, v) -> when(headers.getHeaderString(k)).thenReturn(v)); rest.setHeaders(headers); - Response response = rest.put(S3BUCKET, S3KEY, 0, null, null, 0, null, null, null, null); + Response response = rest.put(S3BUCKET, S3KEY, 0, 0, null, null, null, null); assertEquals(200, response.getStatus()); diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/metrics/TestS3GatewayMetrics.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/metrics/TestS3GatewayMetrics.java index 91df86eab68d..600df053c4ec 100644 --- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/metrics/TestS3GatewayMetrics.java +++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/metrics/TestS3GatewayMetrics.java @@ -310,7 +310,7 @@ public void testCreateKeySuccess() throws Exception { new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); // Create the file keyEndpoint.put(bucketName, keyName, CONTENT - .length(), null, null, 1, null, null, null, body); + .length(), 1, null, null, null, body); body.close(); long curMetric = metrics.getCreateKeySuccess(); assertEquals(1L, curMetric - oriMetric); @@ -322,7 +322,7 @@ public void testCreateKeyFailure() throws Exception { // Create the file in a bucket that does not exist OS3Exception e = assertThrows(OS3Exception.class, () -> keyEndpoint.put( - "unknownBucket", keyName, CONTENT.length(), null, null, 1, null, null, + "unknownBucket", keyName, CONTENT.length(), 1, null, null, null, null)); assertEquals(S3ErrorTable.NO_SUCH_BUCKET.getCode(), e.getCode()); long curMetric = metrics.getCreateKeyFailure(); @@ -358,7 +358,7 @@ public void testGetKeySuccess() throws Exception { new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); // Create the file keyEndpoint.put(bucketName, keyName, CONTENT - .length(), null, null, 1, null, null, null, body); + .length(), 1, null, null, null, body); // GET the key from the bucket Response response = keyEndpoint.get(bucketName, keyName, 0, null, 0, null, null); StreamingOutput stream = (StreamingOutput) response.getEntity(); @@ -465,7 +465,7 @@ public void testCreateMultipartKeySuccess() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); keyEndpoint.put(bucketName, keyName, CONTENT.length(), - null, null, 1, uploadID, null, null, body); + 1, uploadID, null, null, body); long curMetric = metrics.getCreateMultipartKeySuccess(); assertEquals(1L, curMetric - oriMetric); } @@ -474,7 +474,7 @@ public void testCreateMultipartKeySuccess() throws Exception { public void testCreateMultipartKeyFailure() throws Exception { long oriMetric = metrics.getCreateMultipartKeyFailure(); OS3Exception e = assertThrows(OS3Exception.class, () -> keyEndpoint.put( - bucketName, keyName, CONTENT.length(), null, null, 1, "randomId", null, null, null)); + bucketName, keyName, CONTENT.length(), 1, "randomId", null, null, null)); assertEquals(S3ErrorTable.NO_SUCH_UPLOAD.getCode(), e.getCode()); long curMetric = metrics.getCreateMultipartKeyFailure(); assertEquals(1L, curMetric - oriMetric); @@ -521,13 +521,13 @@ public void testCopyObject() throws Exception { new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); keyEndpoint.put(bucketName, keyName, - CONTENT.length(), null, null, 1, null, null, null, body); + CONTENT.length(), 1, null, null, null, body); // Add copy header, and then call put when(headers.getHeaderString(COPY_SOURCE_HEADER)).thenReturn( bucketName + "/" + urlEncode(keyName)); - keyEndpoint.put(destBucket, destKey, CONTENT.length(), null, null, 1, + keyEndpoint.put(destBucket, destKey, CONTENT.length(), 1, null, null, null, body); long curMetric = metrics.getCopyObjectSuccess(); assertEquals(1L, curMetric - oriMetric); @@ -537,7 +537,7 @@ public void testCopyObject() throws Exception { // source and dest same when(headers.getHeaderString(STORAGE_CLASS_HEADER)).thenReturn(""); OS3Exception e = assertThrows(OS3Exception.class, () -> keyEndpoint.put( - bucketName, keyName, CONTENT.length(), null, null, 1, null, null, null, body), + bucketName, keyName, CONTENT.length(), 1, null, null, null, body), "Test for CopyObjectMetric failed"); assertThat(e.getErrorMessage()).contains("This copy request is illegal"); curMetric = metrics.getCopyObjectFailure(); @@ -552,11 +552,11 @@ public void testPutObjectTaggingSuccess() throws Exception { new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); // Create the file keyEndpoint.put(bucketName, keyName, CONTENT - .length(), null, null, 1, null, null, null, body); + .length(), 1, null, null, null, body); body.close(); // Put object tagging - keyEndpoint.put(bucketName, keyName, 0, null, null, 1, null, "", null, getPutTaggingBody()); + keyEndpoint.put(bucketName, keyName, 0, 1, null, "", null, getPutTaggingBody()); long curMetric = metrics.getPutObjectTaggingSuccess(); assertEquals(1L, curMetric - oriMetric); @@ -568,7 +568,7 @@ public void testPutObjectTaggingFailure() throws Exception { // Put object tagging for nonexistent key OS3Exception ex = assertThrows(OS3Exception.class, () -> - keyEndpoint.put(bucketName, "nonexistent", 0, null, null, 1, null, "", + keyEndpoint.put(bucketName, "nonexistent", 0, 1, null, "", null, getPutTaggingBody()) ); assertEquals(S3ErrorTable.NO_SUCH_KEY.getCode(), ex.getCode()); @@ -585,11 +585,11 @@ public void testGetObjectTaggingSuccess() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); keyEndpoint.put(bucketName, keyName, CONTENT - .length(), null, null, 1, null, null, null, body); + .length(), 1, null, null, null, body); body.close(); // Put object tagging - keyEndpoint.put(bucketName, keyName, 0, null, null, 1, null, "", null, getPutTaggingBody()); + keyEndpoint.put(bucketName, keyName, 0, 1, null, "", null, getPutTaggingBody()); // Get object tagging keyEndpoint.get(bucketName, keyName, 0, @@ -620,11 +620,11 @@ public void testDeleteObjectTaggingSuccess() throws Exception { ByteArrayInputStream body = new ByteArrayInputStream(CONTENT.getBytes(UTF_8)); keyEndpoint.put(bucketName, keyName, CONTENT - .length(), null, null, 1, null, null, null, body); + .length(), 1, null, null, null, body); body.close(); // Put object tagging - keyEndpoint.put(bucketName, keyName, 0, null, null, 1, null, "", null, getPutTaggingBody()); + keyEndpoint.put(bucketName, keyName, 0, 1, null, "", null, getPutTaggingBody()); // Delete object tagging keyEndpoint.delete(bucketName, keyName, null, ""); From acde9e70853dcee4fadf6249fc3d5392a3713147 Mon Sep 17 00:00:00 2001 From: peterxcli Date: Sun, 14 Dec 2025 15:18:25 +0800 Subject: [PATCH 6/7] Add Apache License header to S3 Conditional Requests documentation --- .../docs/content/design/s3-conditional-requests.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/hadoop-hdds/docs/content/design/s3-conditional-requests.md b/hadoop-hdds/docs/content/design/s3-conditional-requests.md index e5d5f6469680..9a890680733a 100644 --- a/hadoop-hdds/docs/content/design/s3-conditional-requests.md +++ b/hadoop-hdds/docs/content/design/s3-conditional-requests.md @@ -6,6 +6,19 @@ jira: HDDS-13117 status: draft author: Chu Cheng Li --- + # S3 Conditional Requests Design From ad824b2613e76dbff993b5340528e5b4d0a20431 Mon Sep 17 00:00:00 2001 From: peterxcli Date: Wed, 17 Dec 2025 22:47:54 +0800 Subject: [PATCH 7/7] clarify s3 conditional write will use atomic key rewrite to handle expected etag --- .../content/design/s3-conditional-requests.md | 27 +++++++++++++------ 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/hadoop-hdds/docs/content/design/s3-conditional-requests.md b/hadoop-hdds/docs/content/design/s3-conditional-requests.md index 9a890680733a..c7e517083817 100644 --- a/hadoop-hdds/docs/content/design/s3-conditional-requests.md +++ b/hadoop-hdds/docs/content/design/s3-conditional-requests.md @@ -150,30 +150,41 @@ For failing requests, they still incur the cost of a write RPC and Raft log entr ##### S3 Gateway Layer 1. Parse `If-Match: ""` header. -3. Populate `KeyArgs` with the parsed `expectedETag`. -4. Send the write request (CreateKey/OpenKey) to OM. +2. Populate `KeyArgs` with the parsed `expectedETag`. +3. Send the write request (CreateKey) to OM. -##### OM Layer (Validation Logic) +##### OM Create Phase Validation is performed within the `validateAndUpdateCache` method to ensure atomicity within the Ratis state machine application. 1. **Locking**: The OM acquires the write lock for the bucket/key. 2. **Key Lookup**: Retrieve the existing key from `KeyTable`. 3. **Validation**: - - **Key Not Found**: If the key does not exist, throw `KEY_NOT_FOUND` (maps to S3 412). - - **No ETag Metadata**: If the existing key (e.g., uploaded via OFS) does not have an ETag property, skip ETag validation and allow the operation to proceed. This ensures compatibility with mixed access patterns (OFS and S3A) where S3 Conditional Writes are primarily intended for pure S3 use cases. We do **not** calculate ETag on the spot to avoid performance overhead on the applier thread. + - **No ETag Metadata**: If the existing key (e.g., uploaded via OFS) does not have an ETag property, throw `ETAG_NOT_AVAILABLE` (maps to S3 412). The precondition cannot be evaluated, so we must fail rather than silently proceed. - **ETag Mismatch**: Compare `existingKey.ETag` with `expectedETag`. If they do not match, throw `ETAG_MISMATCH` (maps to S3 412). +4. **Extract Generation**: If ETag matches, extract `existingKey.updateID`. +5. **Create Open Key**: Create open key entry with `expectedDataGeneration = existingKey.updateID`. + +##### OM Commit Phase + +The commit phase reuses the existing atomic-rewrite validation logic from HDDS-10656: + +1. Read open key entry (contains `expectedDataGeneration` set during create phase). +2. Read current committed key from `KeyTable`. +3. Validate `currentKey.updateID == openKey.expectedDataGeneration`. +4. If match, commit succeeds. If mismatch (concurrent modification), throw `KEY_NOT_FOUND` (maps to S3 412). -4. **Execution**: If validation passes, proceed with the operation (adding to OpenKeyTable). +This approach ensures end-to-end atomicity: even if another client modifies the key between Create and Commit phases, the commit will fail. #### Error Mapping | | | | | |---|---|---|---| |**OM Error**|**S3 Status**|**S3 Error Code**|**Scenario**| -|`KEY_GENERATION_MISMATCH`|412|PreconditionFailed|If-None-Match failed| -|`KEY_NOT_FOUND`|412|PreconditionFailed|If-Match failed (key missing)| +|`KEY_ALREADY_EXISTS`|412|PreconditionFailed|If-None-Match failed (key exists)| +|`KEY_NOT_FOUND`|412|PreconditionFailed|If-Match failed (key missing or concurrent modification)| +|`ETAG_NOT_AVAILABLE`|412|PreconditionFailed|If-Match failed (key has no ETag, e.g., created via OFS)| |`ETAG_MISMATCH`|412|PreconditionFailed|If-Match failed (ETag mismatch)| ## AWS S3 Conditional Read Implementation