Core, AWS, REST: Promote the S3 signing endpoint to the main spec #15112

adutra · 2026-01-22T15:50:12Z

Dev ML discussion: https://lists.apache.org/thread/2kqdqb46j7jww36wwg4txv6pl2hqq9w7

This commit promotes the S3 remote signing endpoint from an AWS-specific implementation to a first-class REST catalog API endpoint.

This enables other storage providers (GCS, Azure, etc.) to eventually reuse the same signing endpoint pattern without duplicating the API definition.

OpenAPI Specification changes:

Add /v1/{prefix}/namespaces/{namespace}/tables/{table}/sign/{provider} endpoint to the main REST catalog OpenAPI spec
Define RemoteSignRequest, RemoteSignResult and RemoteSignResponse schemas
Remove the separate s3-signer-open-api.yaml from the AWS module
Update the Python client

Core Module changes (iceberg-core):

Add RemoteSignRequest and RemoteSignResponse model classes, copied from the iceberg-aws module
Add RemoteSignRequestParser and RemoteSignResponseParser for JSON serialization, copied from the iceberg-aws module
Add SIGNER_URI and SIGNER_ENDPOINT properties to CatalogProperties for configuring the signing endpoint
Add V1_TABLE_REMOTE_SIGN field and remoteSign() method to ResourcePaths
Register the new endpoint in Endpoint.java
Add abstract RemoteSignerServlet base class for remote signing tests, copied from the iceberg-aws module

AWS Module changes (iceberg-aws):

Deprecate S3SignRequest and S3SignResponse for removal
Deprecate S3SignRequestParser and S3SignResponseParser for removal
Deprecate S3ObjectMapper for removal
Refactor S3SignerServlet to extend RemoteSignerServlet
Update S3V4RestSignerClient
Move relevant tests to iceberg-core

Dev ML discussion: https://lists.apache.org/thread/2kqdqb46j7jww36wwg4txv6pl2hqq9w7 This commit promotes the S3 remote signing endpoint from an AWS-specific implementation to a first-class REST catalog API endpoint. This enables other storage providers (GCS, Azure, etc.) to eventually reuse the same signing endpoint pattern without duplicating the API definition. OpenAPI Specification changes: - Add `/v1/{prefix}/namespaces/{namespace}/tables/{table}/sign/{provider}` endpoint to the main REST catalog OpenAPI spec - Define `RemoteSignRequest`, `RemoteSignResult` and `RemoteSignResponse` schemas - Remove the separate `s3-signer-open-api.yaml` from the AWS module - Update the Python client Core Module changes (iceberg-core): - Add `RemoteSignRequest` and `RemoteSignResponse` model classes, copied from the iceberg-aws module - Add `RemoteSignRequestParser` and `RemoteSignResponseParser` for JSON serialization, copied from the iceberg-aws module - Add `SIGNER_URI` and `SIGNER_ENDPOINT` properties to `CatalogProperties` for configuring the signing endpoint - Add `V1_TABLE_REMOTE_SIGN` field and `remoteSign()` method to `ResourcePaths` - Register the new endpoint in `Endpoint.java` - Add abstract `RemoteSignerServlet` base class for remote signing tests, copied from the iceberg-aws module AWS Module changes (iceberg-aws): - Deprecate `S3SignRequest` and `S3SignResponse` for removal - Deprecate `S3SignRequestParser` and `S3SignResponseParser` for removal - Deprecate `S3ObjectMapper` for removal - Refactor `S3SignerServlet` to extend `RemoteSignerServlet` - Update `S3V4RestSignerClient`

dimas-b · 2026-01-22T16:30:59Z

open-api/rest-catalog-open-api.yaml

+          $ref: '#/components/responses/AuthenticationTimeoutResponse'
+        503:
+          $ref: '#/components/responses/ServiceUnavailableResponse'
+        5XX:


just wondering: is it valid in Open API to use placeholders like 5xx here?

Yes, the use of 5XX as a status code in OpenAPI specifications is correct and valid:

https://spec.openapis.org/oas/v3.0.3#x4-7-16-2-patterned-fields

dimas-b · 2026-01-22T16:35:43Z

open-api/rest-catalog-open-api.yaml

+      schema:
+        type: string
+        enum:
+          - s3


Will this "lock" generated clients to only allow operating on s3 until the spec is changed? The other parts of this spec do not appear to be bound to S3... I wonder if we could relax this enum to be a free-form string with possible values defined in a way that does not require spec changes to adopt on the client and server sides. WDYT?

Agreed, I hesitated as well. I am OK with a free-form string.

Changed to free-form.

singhpk234 · 2026-01-22T16:50:58Z

open-api/rest-catalog-open-api.yaml

        5XX:
          $ref: '#/components/responses/ServerErrorResponse'

+  /v1/{prefix}/namespaces/{namespace}/tables/{table}/sign/{provider}:


{provide} why do we need that ? a table would ideally be in one object store ? if there are multiple thats fine too, i believe we give absolute path of the uri right ?

I added this, because if/when a catalog server eventually has remote signing available for more than one object storage provider (say, S3 and Azure), it would be good if the server could determine how exactly to sign the request. Without this path parameter, the server would need to apply some heuristics to determine the right object store provider, and hence how to sign the request.

the server would need to apply some heuristics to determine the right object store provider

didn't get this part, we give the path we want to be signed from client to server as part of payload of this request right ? can't we extract that from there (Are you concerned with s3 / s3a / s3n semantics ?)

It's not that easy.

As an example, a request to sign looks like the one below for S3:

PUT /warehouse/db/sales_table/data/date=2024-05/00022-44-55.parquet HTTP/1.1 Host: my-datalake.s3.us-east-1.amazonaws.com Date: Fri, 24 May 2024 12:45:00 GMT Content-Length: 134217728 Content-Type: application/octet-stream

A similar request to GCP would look like:

POST /upload/storage/v1/b/my-datalake-bucket/o?uploadType=media&name=warehouse/db/sales/data/file.parquet HTTP/1.1 Host: storage.googleapis.com Date: Fri, 24 May 2024 12:45:00 GMT Content-Length: 134217728 Content-Type: application/octet-stream

And for Azure:

PATCH /my-container/warehouse/db/sales/data/file.parquet?action=append&position=0 HTTP/1.1 Host: my-datalake.dfs.core.windows.net x-ms-date: Fri, 24 May 2024 12:45:00 GMT x-ms-version: 2023-11-03 Content-Length: 134217728 Content-Type: application/octet-stream

The question is: how do you know the object storage provider so that the server can pick the right signing algorithm? The only (heuristic) way is to inspect the Host header, but that's brittle. It's much simpler if the client tells the server what object storage provider to use.

I am taking of sign request from IRC client to IRC server, i believe what you are showing is IRC server to object store sign ? am i missing something
like IRC client will do a post to /v1/{prefix}/namespaces/{namespace}/tables/{table}/sign with uri as param
https://github.com/apache/iceberg/pull/15112/changes#diff-02549ca620d020dc9ead80088cc14e311e12a69651fa8d394cd41a4308debb2eR4725

i think this would an absolute path right ? s3:////table/data/a.parquet

No, what I'm showing is also what the IRC client sends to the IRC server. IOW, it sends to the IRC server the exact HTTP request it needs to send to the object store. It doesn't send s3:// uris.

interesting, may be wanna then keep this info in the payload ? rather than in the url ? also i wonder how are the location checks happening don't we wanna do prefix match ?

interesting, may be wanna then keep this info in the payload ? rather than in the url, with default if not specified be s3 ?

also i wonder how are the location checks happening don't we wanna do prefix match ?

may be wanna then keep this info in the payload

What do you mean by payload? If you mean in the request body that the signer sends to the signer endpoint, that would require support for request properties, cf dev ML thread:

https://lists.apache.org/thread/gz5nm2xzlhzbc2y3sfossgflnkbm6vq5

But yes, that would work. If people think that's a preferable path, I'm fine with it. (I'm preparing a PR for that as well.)

how are the location checks happening don't we wanna do prefix match ?

Indeed the signer endpoint needs to do a mapping from HTTP URI to S3/GCS/ADLS URIs, in order to validate locations. The mapping can be complex (think: S3 path-style vs virtual-host style, many domains/regions etc.). But it would be even more complex if the signer doesn't even know the target storage provider.

singhpk234 · 2026-01-22T16:52:37Z

open-api/rest-catalog-open-api.yaml

+
+        If remote signing for a specific storage provider is enabled, clients must respect the following configurations when creating a remote signer client:
+         - `signer.uri`: the base URI of the remote signer endpoint. Optional; if absent, defaults to the catalog's base URI.
+         - `signer.endpoint`: the path of the remote signer endpoint. Required. Should be concatenated with `signer.uri` to form the complete URI.


SHOULD or MUST ?

It's complicated 😄

The signer client impl uses org.apache.iceberg.rest.RESTUtil#resolveEndpoint to perform the concatenation of signer.uri and signer.endpoint.

So, signer.endpoint could also be an absolute URL, in which case, signer.uri would be ignored.

I will try to come up with a better wording.

Rephrased, lmk what you think!

singhpk234 · 2026-01-22T16:54:49Z

open-api/rest-catalog-open-api.yaml

          allOf:
            - $ref: '#/components/schemas/Expression'

+    MultiValuedMap:


I believe this is S3Headers eq section in the s3 signer spec ? can we say like ObjectStoreProviderHeader ?

I went for a more generic name because there is nothing specific to remote signing here. This component could perfectly be used for something else in the spec.

adutra · 2026-01-22T18:18:54Z

open-api/rest-catalog-open-api.py

     - `s3.secret-access-key`: secret for credentials that provide access to data in S3
     - `s3.session-token`: if present, this value should be used for as the session token
-     - `s3.remote-signing-enabled`: if `true` remote signing should be performed as described in the `s3-signer-open-api.yaml` specification
+     - `s3.remote-signing-enabled`: if `true` remote signing should be performed as described in the `RemoteSignRequest` schema section of this spec document.


FYI I chose to keep this property specific to S3. I think that even if the signer endpoint is now generic, enablement should be performed for each specific object storage.

you can actually do google GCS cloud access via its s3 gateway; same signing algorithm, just a few different settings to change listing version, endpoint, &c

https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/third_party_stores.md#google-cloud-storage-through-the-s3a-connector

nastra · 2026-01-23T11:40:20Z

aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java

  public String baseSignerUri() {
-    return properties().getOrDefault(S3_SIGNER_URI, properties().get(CatalogProperties.URI));
+    return properties()
+        .getOrDefault(CatalogProperties.SIGNER_URI, properties().get(CatalogProperties.URI));


isn't this breaking existing behavior where one could have provided the s3.signer.uri but now we don't read that property anymore and rely on signer.uri. The same for the endpoint

I don't think so, since the new client introduced in 1.11 is resilient to older servers.

Let's break this into concrete situations:

Server Client Signer config Resulting URI Result

1.10 1.11 s3.signer.uri=...
s3.signer.endpoint=... s3.signer.uri + s3.signer.endpoint ✅

1.10 1.11 s3.signer.uri=... s3.signer.uri + default endpoint ✅

1.10 1.11 s3.signer.endpoint=... catalog URI + s3.signer.endpoint ✅

1.10 1.11 (empty) catalog URI + default endpoint ✅

1.11 1.10 signer.uri=...
signer.endpoint=...
s3.signer.uri=...
s3.signer.endpoint=... s3.signer.uri + s3.signer.endpoint ✅

1.11 1.10 signer.endpoint=...
s3.signer.endpoint=... catalog URI + s3.signer.endpoint ✅

1.11 1.10 signer.uri=...
signer.endpoint=... catalog URI + default endpoint ❌

1.11 1.10 signer.endpoint=... catalog URI + default endpoint ❌

So in summary:

1.11 Clients won't break older servers.

1.11 Servers won't break older clients iif they include both signer.* and s3.signer.* properties for backwards compatibility (which they all should do).

However, once support for the deprecated s3.signer.* properties is removed (1.12), newer clients would break older servers (<= 1.10). If that's concerning, we could e.g. wait a few more minor releases before removal.

shouldn't we be reading S3_SIGNER_URI first and only then fall back to CatalogProperties.SIGNER_URI and then to CatalogProperties.URI?

You are right, I planned for it and forgot to implement 🤦‍♂️

Fixed, and also added a unit test to verify the precedence behavior.

nastra · 2026-01-23T11:40:49Z

aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java

+   * @deprecated since 1.11.0, will be removed in 1.12.0; use {@link CatalogProperties#SIGNER_URI}
+   *     instead.
+   */
+  @Deprecated public static final String S3_SIGNER_URI = CatalogProperties.SIGNER_URI;


I don't think we can just change the value here as that would break backwards compatibility

nastra · 2026-01-23T11:43:13Z

aws/src/test/java/org/apache/iceberg/aws/TestS3FileIOProperties.java

+            "true",
+            CatalogProperties.URI,
+            uri,
+            CatalogProperties.SIGNER_ENDPOINT,


this wasn't needed before but is needed now, which indicates that this is a breaking change for users?

For now it's not required, but it will become required in a future release (1.12 or later).

There is a check + warning here:

https://github.com/adutra/iceberg/blob/f3fc09595bc47d4f0ba7a879d3c7eaf3d70c1e38/aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java#L219-L224

I proactively updated the tests so that they don't break when we make this property required.

nastra · 2026-01-23T11:45:02Z

aws/src/main/resources/s3-signer-open-api.yaml

-
-paths:
-
-  /v1/aws/s3/sign:


I don't think we would want to remove this spec yet. We should probably first deprecate it

nastra · 2026-01-23T11:46:01Z

core/src/main/java/org/apache/iceberg/rest/RESTSerializers.java

    }
  }
+
+  public static class RemoteSignRequestSerializer<T extends RemoteSignRequest>


these all should probably just be package-private and not public

nastra · 2026-01-23T11:48:06Z

aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3SignRequestParser.java

-      gen.writeEndArray();
-    }
-    gen.writeEndObject();
+  public static void headersToJson(


not sure whether we need to make this one and the one below public

github-actions bot added core build AWS OPENAPI labels Jan 22, 2026

adutra force-pushed the promote-sign-endpoint branch from ad95a85 to f3fc095 Compare January 22, 2026 16:00

dimas-b reviewed Jan 22, 2026

View reviewed changes

singhpk234 reviewed Jan 22, 2026

View reviewed changes

adutra commented Jan 22, 2026

View reviewed changes

review

92d733f

nastra reviewed Jan 23, 2026

View reviewed changes

adutra added 4 commits January 25, 2026 18:15

review

6bb3fba

fix support for deprecated signer properties + test

fdd4b22

move constants to RESTCatalogProperties

65f20fd

better handling of deprecation in S3 object mapper

432ca04

Server	Client	Signer config	Resulting URI	Result
1.10	1.11	s3.signer.uri=... s3.signer.endpoint=...	s3.signer.uri + s3.signer.endpoint	✅
1.10	1.11	s3.signer.uri=...	s3.signer.uri + default endpoint	✅
1.10	1.11	s3.signer.endpoint=...	catalog URI + s3.signer.endpoint	✅
1.10	1.11	(empty)	catalog URI + default endpoint	✅
1.11	1.10	signer.uri=... signer.endpoint=... s3.signer.uri=... s3.signer.endpoint=...	s3.signer.uri + s3.signer.endpoint	✅
1.11	1.10	signer.endpoint=... s3.signer.endpoint=...	catalog URI + s3.signer.endpoint	✅
1.11	1.10	signer.uri=... signer.endpoint=...	catalog URI + default endpoint	❌
1.11	1.10	signer.endpoint=...	catalog URI + default endpoint	❌


		paths:

		/v1/aws/s3/sign:

Core, AWS, REST: Promote the S3 signing endpoint to the main spec #15112

Are you sure you want to change the base?

Core, AWS, REST: Promote the S3 signing endpoint to the main spec #15112

Conversation

adutra commented Jan 22, 2026

Uh oh!

dimas-b Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singhpk234 Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nastra Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

dimas-b Jan 22, 2026 •

edited

Loading

singhpk234 Jan 26, 2026 •

edited

Loading

nastra Jan 26, 2026 •

edited

Loading