Missing entries from demozoo

[dag7dev]

What's been included:

• old roms that didn't scraped very well
• newer roms: since the script creation in 2021, newer productions have been added

The usual saulin is missing (it has been already removed in the past).

[avivace]

Some referenced screenshots cannot be found? Could you try running the tests locally to see what breaks @dag7dev ?

[avivace]

I believe we have encoding issues here..

[avivace]

@dag7dev I've also updated the scraper to retain the original demozoo link and parse the release date (when it's complete (dd-mm-yyyy)

[dag7dev]

I believe we have encoding issues here..

is it an issue? they are these characters: ää

@dag7dev I've also updated the scraper to retain the original demozoo link and parse the release date (when it's complete (dd-mm-yyyy)

isnt better yyyy-mm-dd ?

[avivace]

I believe we have encoding issues here..

is it an issue? they are these characters: ää

@dag7dev I've also updated the scraper to retain the original demozoo link and parse the release date (when it's complete (dd-mm-yyyy)

isnt better yyyy-mm-dd ?

sorry yes that's the date that gets set. What I meant was that this happen when the full information is available (day, month year) in the "released" line in demozoo, but not when this is partial (e.g. "2009" or "august 2009" cases are not handled yet)

[dag7dev]

@avivace could we merge this and fix the importers later?

I plan to run them again as soon as I refactor everything, because everything needs to be tested.

[avivace]

The importer needs to be rerun to include the new modifications and this PR has to be updated accordingly. The link to the original release is very important and currently not included in the scraped entries

[avivace]

Rebased against master

[avivace]

@dag7dev I've merged the new entries in 7c9dca0 and f003a0f, however, the scraper/this PR was not in the condition to be merged:

• Encoding is broken. There are a number of game.json with unicode entities escaped and written in the title. Those still need to be fixed
• There was no step checking for duplicates and comparing checksums against the ones we already have. This step flagged 1 duplicate entry
• Your PR didn't include all the new entries that I see running the script now
• Date parsing was missing. This is done. I worked on this and now we have dates, including imprecise ones (only year, year and month and full ones).
• Screenshots are still missing. Only one is taken and we don't navigate to the page containing all of them. We need a script now that can add them back, potentially to the old scraped entries as well.
• There are a number of duplicates which are not flagged by slug similarities and checksum collisions (20-y and pocket tangrams, from what I could see)
• Upstream link was missing, so we were effectively losing the 'source' of those entries. I've added this and we should have a demozoo link as website now.

[avivace]

Also, .gbc files are saved as .gb, for some reason. Not sure if you fixed those manually when committing here or