-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Scanning is no longer idempotent or it possibly never was. After running a verify and a subsequent scan, I recently noticed how the images and verified_at timestamps were removed. I had wondered why artist scans for id 4 would seemingly verify the entire collection again. It is problematic to use the worker that is very much specialized for a single backfill operation. It's not a bad thing to retry the failed lookups but that list will eventually grow indefinitely.
I was able to reproduce this precisely and I have a feeling it comes back to the upsert operation. It's perfect for a single shot bulk insert but the timestamp placeholders create what look like new records instead of look for a whitelist. We only want it to update certain fields, not essentially roll back changes. This is a neat trick and would essentially count as starting over.