Upload of large file fails with 400 Bad Request

[d-blum]

Hey,
thanks for the great api! Listing files, downloading files or uploading small files works fine but I'm facing issue with larger files.
200 MB works fine, but the upload with
410 MB fails with with error message 400 Bad Request

I don't think it's a timeout as the response is returned after only 18 seconds.

Am I missing something? Maybe a property in the header?
I am grateful for all answers. Thanks in advance.

Below you can find the full stacktrace:

2021-11-08 15:45:34 DEBUG PLGSharepointClientOnline:454 - Uploading file 1636386334667my-test.zip to folder /sites/mysite/Shared Documents/test folder/
2021-11-08 15:45:34 DEBUG StreamRestTemplate:147 - HTTP POST https://myurl.com/sites/mysite/_api/web/GetFolderByServerRelativeUrl('/sites/mysite/Shared%20Documents/test%20folder/')/Files/add(url='1636386334667my-test.zip',overwrite=true)
2021-11-08 15:45:34 DEBUG StreamRestTemplate:147 - Accept=[text/plain, application/json, application/*+json, */*]
2021-11-08 15:45:34 DEBUG StreamRestTemplate:147 - Writing [file [/data/jenkins_dev/workspace/DBU_Test_Job/my-test.zip]] as "application/json;odata=verbose"
2021-11-08 15:45:52 DEBUG StreamRestTemplate:147 - Response 400 BAD_REQUEST
org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request
	at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:79)
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122)
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:102)
	at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
	at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:778)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:736)
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:637)
	at com.panxoloto.sharepoint.rest.PLGSharepointClientOnline.uploadFile(PLGSharepointClientOnline.java:471)
	at sharepoint.api.SharepointAPI.uploadFile(SharepointAPI.java:66)
	at sharepoint.api.SharepointAPI.main(SharepointAPI.java:39)

[kikovalle]

I have to look for a way to implement chunked file upload, as there is a limit in the rest API that limits max file upload size in a unique request.

[kikovalle]

@d-blum I've added a uploadBigFiles implementing a chunked file upload that may allow to upload files up to 2Gb to Sharepoint Online. Not yet implemented in the On Premises version.
Notice that the group id of the artifact has also changed as i am planning to generate releases to maven central repository and the change was needed as if not i had to buy a domain.
I tested it with a 200Mb file and the upload is working.

An example call to the method could be this

 int chunkSize = 2048;
JSONObject result = client.uploadBigFile("YourLibraryOrfFolderToUploadWithoutSitePreffix", 
				(Resource) new FileSystemResource(new File("/PATHTOYOURFILE)), metadata, chunkSize);

THe dir (first param) dont include de sites/..., only the path to the list or folder from the site.
metadata works as it worked before, with a json object with metadata to set to the document.
chunkSize is an int with the size of each chunk uploaded to the server.

Let me know if you have any issue with the solution to be able to close this issue.
Thanks

[d-blum]

@kikovalle Unfortunately the new method does not work for me. I'm getting 404 NOT_FOUND. Below is the code I'm using:
client.uploadBigFile(pathToParentfolder, resource, new JSONObject("{}"), 2048);

The log is as follows:
2021-12-09 09:09:14 DEBUG StreamRestTemplate:147 - HTTP POST https://my.domain/sites/My-Site/_api/web/GetFolderByServerRelativeUrl('/sites/My-Site/Shared%20Documents/share/sharepoint-api-java/tests/uploadTests/')/Files/add(url='this%20is%20a%20small%20file.txt',overwrite=true)
2021-12-09 09:09:14 DEBUG StreamRestTemplate:147 - Accept=[text/plain, application/json, application/+json, /]
2021-12-09 09:09:14 DEBUG StreamRestTemplate:147 - Writing [Byte array resource [resource loaded from byte array]] as "application/json;odata=verbose"
2021-12-09 09:09:14 DEBUG StreamRestTemplate:147 - Response 200 OK
2021-12-09 09:09:14 DEBUG StreamRestTemplate:147 - Reading to [java.lang.String] as "application/json;odata=verbose;charset=utf-8"
2021-12-09 09:09:14 DEBUG PLGSharepointClientOnline:418 - Empty file created for chunked file upload
2021-12-09 09:09:14 DEBUG StreamRestTemplate:147 - HTTP POST https://my.domain/sites/My-Site_api/web/getfilebyserverrelativeurl('/sites/My-Site/Shared%20Documents/share/sharepoint-api-java/tests/uploadTests/this%20is%20a%20small%20file.txt')/startupload(uploadId=guid'123a123c-cc3e-49eb-b56d-720291074444')
2021-12-09 09:09:14 DEBUG StreamRestTemplate:147 - Accept=[text/plain, application/json, application/+json, /]
2021-12-09 09:09:14 DEBUG StreamRestTemplate:147 - Writing [[B@df4b72] as "application/octet-stream"
2021-12-09 09:09:14 DEBUG StreamRestTemplate:147 - Response 404 NOT_FOUND

In your project I've discovered the ChunkFileUploader. With the ChunkFileUploader I'm not facing an error 404 NOT_FOUND and the upload works as expected. The code for this approach is:
client.createChunkFileUploader().uploadFile(pathToParentfolder, resource, 5 * 1024 * 1024)

As there is a workaround that works for me I'm good with closing the issue.

[kikovalle]

I have an idea about the cause of the error, let me upload and update as i also detected a bug in last release. Hopefully this afternoon ill be able to commit the fix.

[kikovalle]

Bugfix released with 1.0.4 version. Please let me know if the 404 error is solved for you. Thank you

[d-blum]

I'm still getting the same error with version 1.0.4

[kikovalle]

I've just released 1.0.5 version i hope will solve the issue. Let me know if it is solved, as i see in the exception it was related to the folder name and not to the upload logic itself.

[d-blum]

Unfortunately it still does not work in version 1.0.5

[xsenko]

I'm having this problem with 1.0.8 version.
Uploading small files are fine, but uploading big file with uploadBigFile method returns 404 FILE NOT FOUND