Skip to content

WIP: Implement multi part copy for s3 #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
johha wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

WIP: Implement multi part copy for s3 #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
faa5ebe
Implement missing operations for s3
johha Feb 4, 2026
3efd487
Remove location constraint for AWS us-east-1
johha Feb 4, 2026
606fddc
Print empty properties json when blob does not exist
johha Feb 10, 2026
95402ee
fix error message
johha Feb 10, 2026
d19fc9c
wip: implement multipart copy
johha Feb 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/scripts/s3/assets/cloudformation-s3cli-iam.template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,20 @@
"Effect": "Allow",
"Resource": "arn:aws:logs:*:*:*"
},
{
"Action": ["s3:CreateBucket", "s3:HeadBucket", "s3:DeleteBucket"],
"Effect": "Allow",
"Resource": {
"Fn::Join": [
"",
[
"arn:aws:s3:::",
{ "Ref": "S3Bucket" },
"*"
]
]
}
},
{
"Action": [
"s3:GetObject*",
Expand Down Expand Up @@ -67,6 +81,26 @@
"/*"
]
]
},
{
"Fn::Join": [
"",
[
"arn:aws:s3:::",
{ "Ref": "S3Bucket" },
"*"
]
]
},
{
"Fn::Join": [
"",
[
"arn:aws:s3:::",
{ "Ref": "S3Bucket" },
"*/*"
]
]
}
]
}
Expand Down
7 changes: 4 additions & 3 deletions .github/scripts/s3/run-integration-aws-iam.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,8 @@ pushd "${repo_root}" > /dev/null
--function-name "${lambda_function_name}" \
--zip-file fileb://payload.zip \
--role "${iam_role_arn}" \
--timeout 300 \
--timeout 600 \
--memory-size 512 \
--handler lambda_function.test_runner_handler \
--runtime python3.9

Expand All @@ -58,8 +59,8 @@ pushd "${repo_root}" > /dev/null
tries=0
get_function_status_command="aws lambda get-function --region ${region_name} --function-name ${lambda_function_name}"
function_status=$(${get_function_status_command})
while [[ ( $(echo "${function_status}" | jq -r ".Configuration.State") != "Active" ) && ( $tries -ne 5 ) ]] ; do
sleep 2
while [[ ( $(echo "${function_status}" | jq -r ".Configuration.State") != "Active" ) && ( $tries -ne 15 ) ]] ; do
sleep 3
echo "Checking for function readiness; attempt: $tries"
tries=$((tries + 1))
function_status=$(${get_function_status_command})
Expand Down
1 change: 0 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ Key points
- additional endpoints needed by CAPI still missing
- [Gcs](./gcs/README.md)
- [S3](./s3/README.md)
- additional endpoints needed by CAPI still missing


## Build
Expand Down
36 changes: 19 additions & 17 deletions s3/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,25 @@ The S3 client requires a JSON configuration file with the following structure:

``` json
{
"bucket_name": "<string> (required)",
"credentials_source": "<string> [static|env_or_profile|none]",
"access_key_id": "<string> (required if credentials_source = 'static')",
"secret_access_key": "<string> (required if credentials_source = 'static')",
"region": "<string> (optional - default: 'us-east-1')",
"host": "<string> (optional)",
"port": <int> (optional),
"ssl_verify_peer": <bool> (optional - default: true),
"use_ssl": <bool> (optional - default: true),
"signature_version": "<string> (optional)",
"server_side_encryption": "<string> (optional)",
"sse_kms_key_id": "<string> (optional)",
"multipart_upload": <bool> (optional - default: true),
"download_concurrency": <int> (optional - default: 5),
"download_part_size": <int64> (optional - default: 5242880), # 5 MB
"upload_concurrency": <int> (optional - default: 5),
"upload_part_size": <int64> (optional - default: 5242880) # 5 MB
"bucket_name": "<string> (required)",
"credentials_source": "<string> [static|env_or_profile|none]",
"access_key_id": "<string> (required if credentials_source = 'static')",
"secret_access_key": "<string> (required if credentials_source = 'static')",
"region": "<string> (optional - default: 'us-east-1')",
"host": "<string> (optional)",
"port": <int> (optional),
"ssl_verify_peer": <bool> (optional - default: true),
"use_ssl": <bool> (optional - default: true),
"signature_version": "<string> (optional)",
"server_side_encryption": "<string> (optional)",
"sse_kms_key_id": "<string> (optional)",
"multipart_upload": <bool> (optional - default: true),
"download_concurrency": <int> (optional - default: 5),
"download_part_size": <int64> (optional - default: 5242880), # 5 MB
"upload_concurrency": <int> (optional - default: 5),
"upload_part_size": <int64> (optional - default: 5242880) # 5 MB
"multipart_copy_threshold": <int64> (optional - default: 5368709120) # default 5 GB
"multipart_copy_part_size": <int64> (optional - default: 104857600) # default 100 MB - must be at least 5 MB
}
```
> Note: **multipart_upload** is not supported by Google - it's automatically set to false by parsing the provided 'host'
Expand Down
Loading
Loading