Skip to content

Update to fix some major issues#4

Open
fireattack wants to merge 17 commits intoSraq-Zit:masterfrom
fireattack:contribute
Open

Update to fix some major issues#4
fireattack wants to merge 17 commits intoSraq-Zit:masterfrom
fireattack:contribute

Conversation

@fireattack
Copy link
Collaborator

@fireattack fireattack commented Mar 11, 2023

  • Fixed multi-thread uploading
    • For gigafile, while you can use multi-thread, you MUST finish each chunk thread in order, otherwise gigafile will break your file. And you must initialize and finish the first chunk in single-thread (i.e. wait for it to finish) to set up proper cookies for the other threads to work.
    • To overcome this, I just created a generator and made the thread "stuck" at the end of uploading on file (actually a byte string) reading side until its turn came (to finish). This is to ensure each thread finishes in order (They all start in order, too, but that's obvious). It works, but obviously not the most elegant way. The biggest concern is its memory usage -- it reads the whole chunk into the memory instead of reading from temp file in smaller buffer size. It shouldn't be a concern in general but if you have many threads with large chunk size, it could be on system with limited memory.
  • Add timeout for any requests. Chunk thread will also auto retry if timed out. Can be set with --timeout. Timeout in requests only counts time that no bytes is transferring, so it won't stop long-time running threads. And to my surprise, if you hold the thread by throttling the reading side as I did, it doesn't seem to time out, too.
  • Fixed download filename issue.
  • Added pyproject.toml to make sure PIP install it using modern method. Otherwise it has various Unicode issues for the exe on Windows platform (see On Windows, console script wrapper (exe) created by pip cannot process non-ANSI (Unicode) arguments  pypa/pip#11800 for details).
  • Some other refactoring and QoL changes.

Note: using range header to download in multi-thread is doable, but I usually just use aria2c so I didn't implement it.

@fireattack fireattack changed the title Update to Update to fix some major issues Mar 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant