[ bug ] - The bucket you are attempting to access must be addressed using the specified endpoint

[egarbi]

Description

When trying to sync a bucket in eu-west-1 from a docker running within the same region to a bucket in us-west-1 I get the error mentioned in the title.

The default region set in ~/.aws/config and AWS_REGION and AWS_DEFAULT_REGION are all eu-west-1

To reproduce run

> npx s3p sync 'https://s3.eu-west-1.amazonaws.com/example.documents-assets.eu-west/' 'https://s3.us-west-1.amazonaws.com/example.documents-assets.us-west-1/' --bucket 'example.documents-assets.eu-west' --to-bucket 'example.documents-assets.us-west-1' --verbose --region us-west-1

It will return

parseS3Url: input: :https://s3.eu-west-1.amazonaws.com/example.documents-assets.eu-west/
parseS3Url: input: :https://s3.us-west-1.amazonaws.com/example.documents-assets.us-west-1/
parsed-command-line:
  command: sync
  args:    [] https://s3.eu-west-1.amazonaws.com/example.documents-assets.eu-west/, https://s3.us-west-1.amazonaws.com/example.documents-assets.us-west-1/
  options:
    bucket:   example.documents-assets.eu-west
    toBucket: example.documents-assets.us-west-1
    verbose:  true
    region:   us-west-1
    filter:   ({ LastModified }) => String(LastModified).startsWith('2023-')
    folder:   https://s3.eu-west-1.amazonaws.com/example.documents-assets.eu-west/
    tofolder: https://s3.us-west-1.amazonaws.com/example.documents-assets.us-west-1/

s3p: d: 1s 193μs, items: 0, items/s: 0, listRequests: 2, listWorkers: 2, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)
S3.list-error:
  bucket:     :example.documents-assets.eu-west
  prefix:     undefined
  startAfter: ""
  limit:      1000
  error:      Error:
    class: class S3ServiceException
    stack:
      PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
          at throwDefaultError (/root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/index.js:846:20)
          at /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/index.js:855:5
          at de_CommandError (/root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4734:14)
          at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:484:18
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:138:14
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22

Error:
  class: class S3ServiceException
  stack:
    PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
        at throwDefaultError (/root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/index.js:846:20)
        at /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/index.js:855:5
        at de_CommandError (/root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4734:14)
        at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:484:18
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:138:14
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22

Observations

The same works with s3p 3.4.10

s3p-6f9676c98d-77m84:~# npx s3p@3.4.10 sync --bucket 'example.documents-assets.eu-west' --to-bucket 'example.documents-assets.us-west-1' --verbose --region us-west-1 --filter "js:({ LastModified }) => String(LastModified).startsWith('2023-')"
parsed-command-line: command: sync, options: bucket: example.documents-assets.eu-west, toBucket: example.documents-assets.us-west-1, verbose: true, region: us-west-1, filter: ({ LastModified }) => String(LastModified).startsWith('2023-')
(node:642) NOTE: The AWS SDK for JavaScript (v2) is in maintenance mode.
 SDK releases are limited to address critical bug fixes and security issues only.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the blog post at https://a.co/cUPnyil
(Use `node --trace-warnings ...` to show where the warning was created)
s3p: d: 1s 582μs, items: 0, items/s: 0, listRequests: 2, listWorkers: 4, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)
s3p: d: 2s 565μs, items: 2000, items/s: 999, listRequests: 6, listWorkers: 6, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)
s3p: d: 3s 714μs, items: 5000, items/s: 1666, listRequests: 12, listWorkers: 12, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)
s3p: d: 4s 89ms, items: 5000, items/s: 1222, listRequests: 12, listWorkers: 8, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)
s3p: d: 5s 89ms, items: 9258, items/s: 1819, listRequests: 20, listWorkers: 14, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)

[egarbi]

it seems related to #87

[nathanlisabio]

getting the same error. specifiying region does not fix this problem as shown in #87 .

only by downgrading from latest (3.6) down to (3.4.10) is it able to run properly.

[kriegster108]

For those of you who get failures from NoSuchKey errors and also want cross region syncing I implemented an existing fix for v3.4.10 here: https://github.com/kriegster108/s3p