Weekly build is failing after most recent batch of commits

[dakotahawkins]

https://github.com/shsms/ulysses-annotated/actions/runs/18624344139

I can't see the cause yet but figured an issue was warranted.

[dakotahawkins]

Question about this change in the linked comment: b3bd19f#r168433012

[dakotahawkins]

Do mime's chaiscript eval and eval_file calls need to better report exceptions, maybe like this example? It would be good to have a way to tell what line or content the evaluation chokes on.

https://github.com/shsms/mime/blob/7736ad92b77be8b754444b4a42063565281ef62a/c%2B%2B/src/cmd/chai.cc#L213

[dakotahawkins]

I can't get mime to build, I think the ubuntu latest image might not have gcc 10 on it explicitly

[dakotahawkins]

Here is the change I'd be interested in seeing: dakotahawkins/mime@77d9a57

[shsms]

Hey, thanks for making an issue for this. It is failing because the Joyce Project recently switched to a new website design, and our downloads are failing: https://www.joyceproject.com/1

This is the script that downloads the files: https://github.com/shsms/ulysses-annotated/blob/master/scripts/dl-anno.go If you want to try it locally, it requires the idea crawler server (another old project of mine) to be running: https://github.com/shsms/ulysses-annotated/blob/master/.github/workflows/weekly-release.yml#L31-L42

We need to update the download script to fetch all the text and notes pages and the js files that contain the colour coding information that we use here: https://github.com/shsms/ulysses-annotated/blob/master/scripts/add-anno.mime#L19-L80

If you're able to find the time for this, that would be really amazing. I expect it to take a few days, so if I'm doing it, I will likely need multiple weekends before I can get this running again.

[shsms]

I can't get mime to build, I think the ubuntu latest image might not have gcc 10 on it explicitly

I'll see if I can get it to build again now, but this not the cause of the current issue.

[shsms]

I spent a little bit of time looking at the new website design and it appears to be very javascript heavy. Without having tried anything, I think the html scraping that's currently implemented might not be enough and we need a browser's help.

Idea crawler supports using chrome as the backend, but it is probably using a very old version of the chrome devtools protocol that is not supported by chrome anymore. Not sure.

Otherwise we'll need an alternate download mechanism.

I'll need to do some more research and if it turns out it is not going to be an easy project, I'll email Dr. Hunt who runs The Joyce Project later to let him know of this and ask him if he can find an easy solution for us.

[dakotahawkins]

I'll need to do some more research and if it turns out it is not going to be an easy project, I'll email Dr. Hunt who runs The Joyce Project later to let him know of this and ask him if he can find an easy solution for us.

It looks like it's meant to have an api you can traverse, if you look at network requests and the json responses they seem to include the html source in a single bulk string. It may be worth asking anyway -- they may be generating their json from html files that it would be possible for this project to access in a more straightforward way.

[dakotahawkins]

Here's the project for the webpage itself: https://github.com/alexchunt90/joyce

[lbolla]

It's very easy to download all the notes: the new website provides an API. The list of notes is:

curl https://joyceproject.com/api/notes/

And then, every individual note can be downloaded with, e.g.:

curl https://joyceproject.com/api/notes/TU-bJZUBeQi-aLQfd6-b

What I am not sure about is how to attach notes to the original text.
But scraping may not be required at all, which would simplify things considerably.

[lbolla]

Otherwise, https://joyce-staging.net/ seems to have the old website version.

[shsms]

Otherwise, https://joyce-staging.net/ seems to have the old website version.

Oh wow, I had no idea. Thanks for pointing that out! I just wrote to Prof. Hunt asking if that is going to available long term. If so, it would be a very easy fix indeed.

[alexchunt90]

Hi all, dev for the Joyce Project here, sorry for the disruption! I left the old PHP site up under joyce-staging.net for situations like this one, but we don't plan to maintain it long term.

There's unfortunately no way to access notes as HTML files on the new site, they're stored in Elasticsearch on the backend. I'd recommend using the API. There's also an endpoint for chapters and media, which stores image captions.

curl https://joyceproject.com/api/chapters/
curl https://joyceproject.com/api/media/

I have been working on a way to batch process all documents to static HTML files, but I'm not sure how soon I'll be able to ship it.

[alexchunt90]

The documents store links like so:

<a href=\"Z0-bJZUBeQi-aLQfd7Cb\" data-color=\"0000FF\" data-tag=\"\" data-type=\"annotation\">

The href refers to the ID of a note file, it'd need to be modified for a static site. Whenever I implement batch export, I plan to modify it to something like:

<a href=\"/notes/Z0-bJZUBeQi-aLQfd7Cb.htm\" data-color=\"0000FF\" data-tag=\"\" data-type=\"annotation\">

[lbolla]

Thanks! I have hacked together a script that downloads chapters and notes and creates an epub. You may take inspiration from it if you want: https://github.com/lbolla/ulysses