Skip to content

Conversation

@gowrisankar22
Copy link
Contributor

@gowrisankar22 gowrisankar22 commented Dec 16, 2025

  1. Problem

  2. The BOSH S3 CLI currently uses AWS SDK V2 defaults for S3 uploads and downloads, which are significantly lower than other CLI tools, resulting in suboptimal performance example for https://docs.cloud.google.com/distributed-cloud/docs

  • Default Upload/Download Concurrency: 5
  • Default Upload/Download Part Size: 5MB

https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/s3/manager#Downloader

These conservative defaults were designed for general-purpose use cases but create a bottleneck for BOSH operations that require higher throughput. Unlike the AWS CLI, which offers tunable parameters, there was no way for users to optimize transfer performance based on their specific infrastructure requirements and network conditions.

Solution
This PR makes S3 transfer concurrency and part size configurable, allowing operators to:

  • Increase parallel transfer threads to match available network bandwidth and client machine capabilities
  • Optimize multipart upload/download sizes for specific network conditions and storage backend characteristics
  • Enable throughput optimization without being constrained by conservative SDK defaults.

These properties are purely optional and users can choose if they really need to change the default behavior. At least for google's new air-gapped offering we needed. https://docs.cloud.google.com/distributed-cloud/docs

  1. Problem
    AWS SDK Compatibility (CRC64NVME): The latest bosh-s3cli uses the newest AWS SDK V2, which defaults to x-amz-sdk-checksum-algorithm:CRC64NVME. This has caused compatibility issues to GDCH.

Solution
In the interim, we have to disable this behavior using the configurable parameter disable_checksums.

Minor improvement --> fixing the pascalcase for RequestChecksumCalculationEnabled and UploaderRequestChecksumCalculationEnabled

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Dec 16, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

@gowrisankar22 gowrisankar22 force-pushed the s3cli-concurrency-config branch from d0b2b11 to c172bca Compare December 16, 2025 11:47
@gowrisankar22 gowrisankar22 marked this pull request as ready for review January 28, 2026 11:36
@gowrisankar22 gowrisankar22 force-pushed the s3cli-concurrency-config branch from c14e35a to 2f102dd Compare January 28, 2026 11:46
@gowrisankar22 gowrisankar22 changed the title (feat): make S3 transfer concurrency and part size configurable (feat): make S3 transfer concurrency, part size, checksum configurable Jan 28, 2026
@github-project-automation github-project-automation bot moved this from Inbox to Waiting for Changes | Open for Contribution in Foundational Infrastructure Working Group Jan 28, 2026
@beyhan
Copy link
Member

beyhan commented Jan 29, 2026

FYI: Thanks for the contribution! Just wanted to let you know we're actually moving this functionality to https://github.com/cloudfoundry/storage-cli. That one is already adding concurrency support. Maybe it will be better to first contribute there and cherry pick back to this.

@gowrisankar22
Copy link
Contributor Author

FYI: Thanks for the contribution! Just wanted to let you know we're actually moving this functionality to https://github.com/cloudfoundry/storage-cli. That one is already adding concurrency support. Maybe it will be better to first contribute there and cherry pick back to this.

cloudfoundry/storage-cli#29

Copy link

@a-hassanin a-hassanin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes!

@github-project-automation github-project-automation bot moved this from Waiting for Changes | Open for Contribution to Pending Merge | Prioritized in Foundational Infrastructure Working Group Feb 4, 2026
@a-hassanin a-hassanin requested review from a team, a-b and evanfarrar and removed request for a team February 4, 2026 09:59
@beyhan beyhan merged commit c89b2e7 into cloudfoundry:main Feb 5, 2026
7 checks passed
@github-project-automation github-project-automation bot moved this from Pending Merge | Prioritized to Done in Foundational Infrastructure Working Group Feb 5, 2026
@gowrisankar22 gowrisankar22 deleted the s3cli-concurrency-config branch February 6, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

3 participants