Skip to content

Conversation

@gowrisankar22
Copy link
Contributor

same changes as cloudfoundry/bosh-s3cli#61

  1. Problem

  2. The BOSH S3 CLI currently uses AWS SDK V2 defaults for S3 uploads and downloads, which are significantly lower than other CLI tools, resulting in suboptimal performance example for https://docs.cloud.google.com/distributed-cloud/docs

  • Default Upload/Download Concurrency: 5
  • Default Upload/Download Part Size: 5MB

https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/s3/manager#Downloader

These conservative defaults were designed for general-purpose use cases but create a bottleneck for BOSH operations that require higher throughput. Unlike the AWS CLI, which offers tunable parameters, there was no way for users to optimize transfer performance based on their specific infrastructure requirements and network conditions.

Solution
This PR makes S3 transfer concurrency and part size configurable, allowing operators to:

  • Increase parallel transfer threads to match available network bandwidth and client machine capabilities
  • Optimize multipart upload/download sizes for specific network conditions and storage backend characteristics
  • Enable throughput optimization without being constrained by conservative SDK defaults.

These properties are purely optional and users can choose if they really need to change the default behavior. At least for google's new air-gapped offering we needed. https://docs.cloud.google.com/distributed-cloud/docs

  1. Problem
    AWS SDK Compatibility (CRC64NVME): The latest bosh-s3cli uses the newest AWS SDK V2, which defaults to x-amz-sdk-checksum-algorithm:CRC64NVME. This has caused compatibility issues to GDCH.

Solution
In the interim, we have to disable this behavior using the configurable parameter disable_checksums.

Minor improvement --> fixing the pascalcase for RequestChecksumCalculationEnabled and UploaderRequestChecksumCalculationEnabled

@github-project-automation github-project-automation bot moved this from Inbox to Waiting for Changes | Open for Contribution in Foundational Infrastructure Working Group Jan 30, 2026
Copy link
Contributor

@serdarozerr serdarozerr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, just some minor suggestions

johha
johha previously approved these changes Feb 3, 2026
Copy link

@a-hassanin a-hassanin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-project-automation github-project-automation bot moved this from Waiting for Changes | Open for Contribution to Pending Merge | Prioritized in Foundational Infrastructure Working Group Feb 4, 2026
@a-hassanin a-hassanin merged commit 13260ff into cloudfoundry:main Feb 4, 2026
3 of 7 checks passed
@github-project-automation github-project-automation bot moved this from Pending Merge | Prioritized to Done in Foundational Infrastructure Working Group Feb 4, 2026
@gowrisankar22 gowrisankar22 deleted the s3cli-support-concurrency branch February 4, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

6 participants