Current file.save() function unsuitable for large files #61
Description
Please let me know if I'm just saving things wrong or inefficiently. If I'm simply doing it wrong, I apologize.
At present, the file.save() functionality is unsuitable for large files, seemingly using multiple times more RAM than the actual size of the file itself, and on large enough files it will simply loop forever.
The snippet I'm using to save files is:
f = open(fileNamePath, "ab")
currentFile.save(f)
f.close() (just to be clear, fileNamePath is a string I have already set correctly, where the directory it will be saving to exists, but the file itself does not exist yet, hence opening it in append mode)
On a system with 16GB of ram, far more swap than necessary, and virtually nothing else running, absurdly large files like https://www.moddb.com/games/world-of-warcraft/downloads/patch-3-3-x-to-4-0-0-eu-full (5.21gb) or https://www.moddb.com/downloads/patch-3-3-5-to-4-0-0-us-full (4.71gb) will simply continually max out ram, fall back down to a lower amount of ram, and repeat forever, without ever saving the file.
Some less large files, such as https://www.moddb.com/games/heroes-vi/downloads/patch-2-1 (2.13gb) will eventually save, but will use multiple times more RAM than the size of the file itself before saving, requiring any other applications to be closed and your computer briefly freezing/slowing at the peak of RAM usage (I believe it also took a few repeats of RAM buildup before it saved, but I can't remember):
Looking at a large but still not outrageously large file such as https://www.moddb.com/games/panzer-corps/downloads/patch-1-14 (857 mb) shows that it seems to eat up to 11 times the size of the file itself in RAM before it saves:
To be clear, I completely understand if the official stance of the tool would be that a 5gb moddb file is absurd and should be downloaded manually. If this is the case, I would rather just have some sort of failsafe or warning/error message in the program, so that it will not continually loop forever and/or freeze your computer. Beyond that, needing almost 10gb of ram to save a 857mb file does seem like a legitimate issue.
As mentioned above, please let me know if I'm simply using this function wrong. For all I know, there's a more memory-efficient way to save this to a file without modifying your moddb api itself.