Optimize performance with parallel processing and faster compression settings #3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…settings - Add parallel image processing using threading (default: CPU core count) - Convert JPEG XL and WebP simultaneously for each image - Reduce JPEG XL effort from 9 to 7 (faster with minimal size difference) - Reduce WebP method from 6 to 4 (faster with minimal size difference) - Skip files already in optimized formats (JXL/WebP) - Add --workers flag to control parallelism - Update README with performance optimizations documentation Expected speed improvements: 4-8x faster overall processing time
The exc_info=True parameter only works within an active exception handler. Moved exception logging with full traceback to worker thread where exceptions occur, and removed exc_info=True from main thread where we only have error message strings.
The multi-threaded path was missing hang detection that exists in the single-threaded path. Added timeout to result_queue.get() and implemented hang detection logic with notifications to match single-threaded behavior. - Add timeout to result_queue.get() to prevent indefinite blocking - Implement hang detection checking every 60 seconds - Identify and report stuck images when no progress for 5+ minutes - Send notifications matching single-threaded path behavior
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements several performance optimizations to significantly speed up image processing, along with critical bug fixes.
Changes
--workersflag to control parallelismPerformance Improvements
Bug Fixes
exc_info=Trueto worker thread where exception context existsTesting
Documentation
--workersflag