Skip compression of already compressed files based on saved file hashes

[PetrDlouhy]

This is better approach to fix #279 than MR #295.
It saves hashes of all files and compresses only files, their hash changed.

[adamchainz]

Can’t this be done with an existence check rather than an extra file of metadata? e.g. if trying to compress app.abc123.js, avoid gizp compressing if app.abc123.js.gz exists, and likewise if .br exists.

[PetrDlouhy]

@adamchainz It would be needed to regenerate the compressed file if the app.abc123.js changes.
Maybe it could be determined by the modification dates of the files, but I am not sure, if that mechanism couldn't fail in some cases. I just felt, that the MD5 sums are more reliable.

[adamchainz]

The hash abc123 in the filename is already an MD5 hash. I'm not sure you've followed what Whitenoise is already doing?

[PetrDlouhy]

In my project, I see compressed versions for files with hash as well as files without hash. For example for base.css the manage.py collecstatic creates all these files:

base.css
base.css.br
base.css.gz
base.1f418065fc2c.css
base.1f418065fc2c.css.br
base.1f418065fc2c.css.gz

So if I haven't overlooked something, it means, that the problem is still valid for half of the files.

[adamchainz]

Right - those unhashed files are “left over” by ManifestStaticFilesStorage. I’m not sure it’s even sensible to leave those files around - in normal use of staticfiles they won't be referenced. Whitenoise has the WHITENOISE_KEEP_ONLY_HASHED_FILES setting to remove them, which I've turned on for many projects as a nice boost. I am considering making that the default, or even proposing it for Django.

[PetrDlouhy]

@adamchainz OK, thank you for explaining. That is very neat option.
Then skipping compression of already compressed files should be as easy as this: #347