Skip to content

Conversation

@qxprakash
Copy link
Collaborator

this fixes #6167

AI summary :

The Bug: S3 Multipart Upload Fails with InvalidPart

What happened
When uploading large files (200MB+) from Google Drive → Companion → S3 using multipart upload, the upload would reach ~96-100% and then fail with S3's InvalidPart error on CompleteMultipartUpload.

Root cause
A version mismatch between two AWS SDK packages caused by yarn dedupe.

Commit 3c3034b40 ("Dedupe dependencies #6085") ran yarn dedupe. Companion had @aws-sdk/client-s3 and @aws-sdk/lib-storage both at "^3.338.0", which previously both resolved to 3.600.0. But the @uppy/transloadit package elsewhere in the monorepo had "@aws-sdk/client-s3": "^3.891.0". Dedupe merged the resolutions — client-s3 jumped to 3.896.0 while lib-storage stayed at 3.600.0.

Why the mismatch matters
AWS SDK client-s3@3.896.0 introduced a breaking behavioral change in its checksum middleware:
Old (3.600.0): Default checksum = MD5, no auto-injection. UploadPart requests sent no checksum header.
New (3.896.0): Default checksum = CRC32, requestChecksumCalculation defaults to "WHEN_SUPPORTED". The middleware auto-injects x-amz-checksum-crc32 on every UploadPart request.

The old lib-storage@3.600.0 didn't know about this new behavior. It called CreateMultipartUpload without setting ChecksumAlgorithm. Then client-s3@3.896.0 added CRC32 checksums to each UploadPart. S3 saw parts with checksums that weren't declared at creation time → rejected with InvalidPart at CompleteMultipartUpload.

How the fixes work

Option A — requestChecksumCalculation: 'WHEN_REQUIRED' (workaround):
Tells the new client-s3 to only add checksums when S3 requires them (which it doesn't for UploadPart). This restores the old behavior — no checksum headers, no mismatch.

Option B — Upgrade all AWS SDK packages to ^3.986.0 (proper fix):
The new lib-storage@3.986.0 is aware of the CRC32 checksum behavior. When it detects requestChecksumCalculation = "WHEN_SUPPORTED", it explicitly sets ChecksumAlgorithm: "CRC32" on the CreateMultipartUpload call. Now S3 expects CRC32 checksums on the parts, the parts have CRC32 checksums → everything is consistent → upload succeeds.

TL;DR
yarn dedupe upgraded client-s3 but not lib-storage. The new client-s3 started adding CRC32 checksums to parts, but the old lib-storage didn't declare CRC32 at multipart creation time. S3 rejected the mismatch. Fix: upgrade both packages together so they agree on checksum behavior.

@changeset-bot

This comment was marked as resolved.

@qxprakash qxprakash requested review from Murderlon and mifi February 11, 2026 12:39
Copy link
Contributor

@mifi mifi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice finding! This change doesnt affect customers (only uppy development) so no changeset needed

Copy link
Member

@Murderlon Murderlon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing a changeset as well

Comment on lines +37 to +41
"@aws-sdk/client-s3": "^3.986.0",
"@aws-sdk/client-sts": "^3.986.0",
"@aws-sdk/lib-storage": "^3.986.0",
"@aws-sdk/s3-presigned-post": "^3.986.0",
"@aws-sdk/s3-request-presigner": "^3.986.0",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want to pin them

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

remote uploads looping in @uppy/companion with aws-s3 uploader

3 participants