Multipart s3 upload failing

[helrond]

Two users are reporting that attempts to upload specific bags to S3 are failing with the following error: "S3Error: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed." This error occurs consistently with these bags (which are about 2GB) but does not occur with all bags over the S3 multipart upload threshold. The bags in question all have at least one large .mov file which by itself is over the multipart threshold. These two users have reproduced this problem on multiple machines and using different network connections.

I have not been able to reproduce this error on my end, either by uploading the tar file produced by DART or by re-creating the bag with the original files from scratch.

Any ideas of where to go next with troubleshooting?

[ntallman]

Is there anything unusual about the bag name, like a non-printing character? We may not be able to sort this one out until we've hired a new Lead Developer and they're up-to-speed. I can ask Andrew to look, but his availability is intermittent. At least it only affects the upload portion, so you could use other tools like CyberDuck to get it into the bucket. One other thought, is there sufficient memory on the workstations that fail with this upload? The chunks are stored in memory on their route to S3, if there isn't sufficient memory, it might impact the upload.

[helrond]

Thanks for the quick response @ntallman - we're planning on working around this at the moment with a secondary upload tool, as you suggested. But I'd still like to get to the bottom of this when we can.

I checked on the memory allocation, and while one workstation only had 8GB, the other had 64, so I don't think that one's running out of memory.

[diamondap]

DART 3 has a more robust uploader. The new version isn't out yet, but we may be able to close this when that release becomes available.