Skip to content

Scan acts as it just inserted brand new records #57

@w0rd-driven

Description

@w0rd-driven

Scanning is no longer idempotent or it possibly never was. After running a verify and a subsequent scan, I recently noticed how the images and verified_at timestamps were removed. I had wondered why artist scans for id 4 would seemingly verify the entire collection again. It is problematic to use the worker that is very much specialized for a single backfill operation. It's not a bad thing to retry the failed lookups but that list will eventually grow indefinitely.

I was able to reproduce this precisely and I have a feeling it comes back to the upsert operation. It's perfect for a single shot bulk insert but the timestamp placeholders create what look like new records instead of look for a whitelist. We only want it to update certain fields, not essentially roll back changes. This is a neat trick and would essentially count as starting over.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions