Duplicates of JPG derivatives not detected

[DominicBM]

I am observing some problems with the script's duplicate detection when it comes to the JPG derivatives, either because conversion does not always produce exactly identical outputs, or because some previous bot runs were using slightly different settings, (or it's just not looking for duplicates of the JPGs after a conversion from TIFF).

This is fine when the names are the same, and overwriting an older version of a file with another does no harm (or even is good, because it enforces consistency).

But when the bot is also moving page names at the same time, it will move the TIFF, leave the old JPG in place because it did not detect it, and then upload a new JPG (now not linked form the TIFF), creating a bit of a mess.

Example:

https://commons.wikimedia.org/wiki/File:Grand_Canyon._Same_locality_as_433._Old_Nos._470,_473,_500_-_NARA_-_517801.tif
https://commons.wikimedia.org/wiki/File:Grand_Canyon._Same_locality_as_433._Old_Nos._470,_473,_500_-_NARA_-_517801.jpg
https://commons.wikimedia.org/wiki/File:Grand_Canyon._Same_locality_as_433._Old_Nos._470,_473,_500,_1871_-_1878_-_NARA_-_517801.jpg

[DominicBM]

At the very least, will have to go through the bot's move log and fix a few dozen of these manually, by requesting deletion of the old JPG and fixing links on the new one.

https://commons.wikimedia.org/w/index.php?title=Special%3ALog&type=move&user=US_National_Archives_bot&page=&year=&month=-1&tagfilter=&hide_thanks_log=1

Not sure if there is a good way to prevent this in the future.

[jwestgard]

To be honest, I haven't really delved into the renaming functions, but my understanding was that the focus was on changing the names to the canonical ones more than dealing with the JPG derivatives. Since the JPG is being treated as something of a side-effect to the uploading of the TIFF, I can totally imagine that there might be a flaw in the logic there. If it is causing problems, my first inclination would be to turn it off altogether, so it doesn't make a mess.

[DominicBM]

It's causing a mess because of duplication, but I'm not that worried overall, because I see these TIFF-to-JPG conversion products as temporary. Eventually, we will remove it (and all the JPG versions) when TIFFs are fully supported in MediaWiki. However, as I'm actually catching quite a few files that are getting renamed, I think that is a sign that the renaming is actually a useful feature for enforcing consistency and cleaning up our old uploads.

Probably the best workflow for now is if a TIFF is being moved, automatically move the associated JPG file (i.e., if there is a file with the same file name, but different extension, move it), and then always upload the JPG derivative over the old one. (Rather than separately looking for a JPG duplicate based on the derivative, since that's not catching them.) I think this is not a huge issue, though.