Decompress object streams asynchronously when it's possible #20601
Conversation
|
/botio test |
From: Bot.io (Linux m4)ReceivedCommand cmd_test from @calixteman received. Current queue size: 0 Live output at: http://54.241.84.105:8877/3e2fd092ee6f1f3/output.txt |
From: Bot.io (Windows)ReceivedCommand cmd_test from @calixteman received. Current queue size: 0 Live output at: http://54.193.163.58:8877/e7b88f8a76dd3a1/output.txt |
timvandermeij
left a comment
timvandermeij
left a comment
There was a problem hiding this comment.
r=me, with the comment addressed and passing tests. Thanks!
From: Bot.io (Linux m4)FailedFull output at http://54.241.84.105:8877/3e2fd092ee6f1f3/output.txt Total script time: 42.07 mins
Image differences available at: http://54.241.84.105:8877/3e2fd092ee6f1f3/reftest-analyzer.html#web=eq.log |
From: Bot.io (Windows)FailedFull output at http://54.193.163.58:8877/e7b88f8a76dd3a1/output.txt Total script time: 146.76 mins
Image differences available at: http://54.193.163.58:8877/e7b88f8a76dd3a1/reftest-analyzer.html#web=eq.log |
calixteman
force-pushed
the
decompress_obj_stream
branch
2 times, most recently
from
7399e09 to
a60f53f
Compare
|
/botio test |
From: Bot.io (Windows)ReceivedCommand cmd_test from @calixteman received. Current queue size: 0 Live output at: http://54.193.163.58:8877/4d22a147cc5d465/output.txt |
From: Bot.io (Linux m4)ReceivedCommand cmd_test from @calixteman received. Current queue size: 0 Live output at: http://54.241.84.105:8877/3def7312905b772/output.txt |
From: Bot.io (Linux m4)FailedFull output at http://54.241.84.105:8877/3def7312905b772/output.txt Total script time: 43.01 mins
Image differences available at: http://54.241.84.105:8877/3def7312905b772/reftest-analyzer.html#web=eq.log |
From: Bot.io (Windows)FailedFull output at http://54.193.163.58:8877/4d22a147cc5d465/output.txt Total script time: 56.13 mins
Image differences available at: http://54.193.163.58:8877/4d22a147cc5d465/reftest-analyzer.html#web=eq.log |
Most of the time, the object streams are compressed using FlateDecode (and in future with BrotliDecode). So in order to improve the performances we can decompress those streams with a built-in decompressor but it has to be done asynchronously. Since it cannot be done when fetching which is synchronous, we need to do it as part of the PDF parsing process. The drawback is that this requires more memory since we need to keep both the compressed and uncompressed versions of the object streams in memory until the parsing is done.
calixteman
force-pushed
the
decompress_obj_stream
branch
from
a60f53f to
e7900fc
Compare
|
/botio test |
From: Bot.io (Linux m4)ReceivedCommand cmd_test from @calixteman received. Current queue size: 0 Live output at: http://54.241.84.105:8877/4828dfa2ddb9125/output.txt |
From: Bot.io (Windows)ReceivedCommand cmd_test from @calixteman received. Current queue size: 0 Live output at: http://54.193.163.58:8877/b235e05cf63ad98/output.txt |
From: Bot.io (Windows)FailedFull output at http://54.193.163.58:8877/b235e05cf63ad98/output.txt Total script time: 59.57 mins
Image differences available at: http://54.193.163.58:8877/b235e05cf63ad98/reftest-analyzer.html#web=eq.log |
From: Bot.io (Linux m4)FailedFull output at http://54.241.84.105:8877/4828dfa2ddb9125/output.txt Total script time: 60.00 mins |
3 participants
Most of the time, the object streams are compressed using FlateDecode (and in future with BrotliDecode).
So in order to improve the performances we can decompress those streams with a built-in decompressor but it has to be done asynchronously. Since it cannot be done when fetching which is synchronous, we need to do it as part of the PDF parsing process.
The drawback is that this requires more memory since we need to keep both the compressed and uncompressed versions of the object streams in memory until the parsing is done.