[New Notebook]: Example harvest json-ld from ERDDAP or other sources

[MathewBiddle]

Programming Language

Python

Module/Software/IOOS data service?

ERDDAP, schema.org, extruct?

Module/Software/IOOS data service?

Yes, see this gist - https://gist.github.com/MathewBiddle/c5fee0741b0d94659bb98b0ad67f1eb9

Provide a description.

Many data repositories have included the schema.org specification in their dataset landing pages for Google Dataset Search. Additionally, ERDDAP provides json-ld markup in the info/index.html for the entire ERDDAP as well as each individual dataset landing page (https://erddap.github.io/docs/server-admin/additional-information?_highlight=json#json-ld). I'd like to create an example of harvesting these metadata so we can make a map of datasets. We can test with datasets in the gist above as well as the IOOS ERDDAP https://erddap.ioos.us/erddap/index.html

[MathewBiddle]

I think some pages do some funky stuff that hides the json-ld, so we might need to use something like selenium to trick the call as a browser.

See https://opc.dataone.org/view/doi%3A10.25494%2FP6901R. When I hit it using requests , the node for "script", {"type":"application/ld+json"} doesn't exist. However, when I open the webpage and 'inspect' it, I can see the appropriate node in the html.