Use chunked upload(?)

[DominicBM]

I had thought that the script was using chunked uploading, but since I see that chunk is a non-default API parameter, I think that this would fix the "broken pipe" errors I am still getting on large files. The Commons page says that chunked upload should allow files up to 1 GB, so these ~200 MB files, while large, shouldn't be killing the script.

[DominicBM]

Confirmed that it's breaking on anything over 100MB, which is the non-chunk maximum file size, so this is blocking several files from working (and killing the upload run when it hits one).

[jwestgard]

There is a "max-size" parameter for the upload bot. This is currently hardcoded as "None", but that could perhaps be overriden in the command parameters (I'd have to double check whether that's already the case or not). If max size is set, however, all that seems to happen is that the file is set aside into an "overflow directory", and the metadata is saved to a text file. As far as I can tell, there is no attempt in the current code to switch over to chunked uploading.

That said, if we set a max size at 100 MB, at least the script wouldn't fail upon encountering a large file. We could then continue to upload smaller files while we look toward implementing chunking.

[DominicBM]

Yes, I do remember that now, thanks. I'll use that parameter from now on.

The original script was written before chunked uploading existed, but I thought Fran had since added that. It would be good to look into that.