list of contributions

[xrotwang]

My ideal for the list of contributions to be included in CrossGram would be a simple list of Zenodo DOIs for (particular versions of) datasets. (I have some code - to be included in cldf-zenodo - to fetch such datasets from Zenodo.)
The CrossGram app would then function mostly as (selective) catalogue of CLDF datasets on Zenodo, augmented with as much visualization as is possible generically.

[johenglisch]

Yeah, that makes sense.

Just for my own brain's sake, this is how I imagine this to happen:

1. CrossGram editors specify a Zenodo DOI in contributions.json (as opposed to a folder name)
2. clld initdb downloads the data into a cache folder
3. clld initdb then loads the CLDF data into the database
4. The cache folder is kept around, so one can repopulate the database without re-downloading the same stuff over and over again

Although I would still keep the option around to include local repositories (for testing unpublished datasets in a local instance).

[xrotwang]

What about something like this:

1. initializedb.main prompts for a contribution.json location - defaulting to the one in the repos.
2. Items in contribution.json are either DOIs or directory names (relative to the parent of contribution.json).

Or maybe we hardwire two contributions.json locations and switch between them with a internal switch like with dictionaria.

[xrotwang]

Actually, we might want to push the analogy with dictionaria even further: I.e. have separate repository for the editorial backend - even though right now this might just look like

├── README.md
├── submissions
│   └── contributions.json
├── submissions-internal
│   └── contributions.json

I'd guess we might want to include at least a short textual description with each dataset, so this may be additional content here.

[johenglisch]

Sounds good. We could go even more analogous and split the contributions.json into per-submission metadata files:

|-- README.md
|-- submissions
|   |-- dryerorder
|       |-- intro.md
|       |-- md.json
|-- submissions-internal
|   |-- haspelmathcomparison
|       |-- intro.md
|       |-- md.json

[xrotwang]

True. I'm a bit torn, though, because I wouldn't want CrossGram to add information to datasets, which should be already included in the dataset on Zenodo. But then, I can already think of

• DOI
• publication data

and possibly

• responsible editor

[xrotwang]

https://github.com/crossgram/crossgram-internal

[johenglisch]

Yeah, it definitely should not contain the wealth of information that Dictionaria's metadata files provide.

I wouldn't make the metadata much more elaborate than the entries in the current contributions.json:

    {
        "id": "haspelmathcomparison",
        "number": 1,
        "published": "2020-09-08",
        "repo": "../../cldf-datasets/haspelmathcomparison",
        "authors": [
            "Martin Haspelmath",
            "The Comparative Constructions Team"
        ]
    }

(authors and repo will probably be replaced by doi once the dataset is published)

[xrotwang]

Yes. I'm not really happy with the path for repo - but also wouldn't want to go as far as using git submodules. Maybe we should have submissions-internal/datasets/ (hidden via .gitignore) and repo must be a subdirectory name in there?

[xrotwang]

Or actually have datasets in the repos root and use this as cache dir for the DOIs, too.

[johenglisch]

Yup, that was what I was about to suggest, too.

[xrotwang]

Btw. here's my code to download from zenodo (we might want to include it here, until I get around to finishing the cldf-zenodo package):

import io
import re
import json
import pathlib
import zipfile
import urllib.request

from bs4 import BeautifulSoup as bs
import requests


def download_from_doi(doi, outdir=pathlib.Path('.')):
    res = requests.get('https://doi.org/{0}'.format(doi))
    assert re.search('zenodo.org/record/[0-9]+$', res.url)
    res = requests.get(res.url + '/export/json')
    soup = bs(res.text, 'html.parser')
    res = json.loads(soup.find('pre').text)
    assert any(kw.startswith('cldf:') for kw in res['metadata']['keywords'])
    for f in res['files']:
        if f['type'] == 'zip':
            r = requests.get(f['links']['self'], stream=True)
            z = zipfile.ZipFile(io.BytesIO(r.content))
            z.extractall(str(outdir))
        elif f['type'] == 'gz':
            # what about a tar in there?
            raise NotImplementedError()
        elif f['type'] == 'gz':
            raise NotImplementedError()
        else:
            urllib.request.urlretrieve(
                f['links']['self'],
                outdir / f['links']['self'].split('/')[-1],
            )
    return outdir

The resulting directory can then be searched for datasets using pycldf.iter_datasets.

[johenglisch]

Huh… pycldf.iter_datasets seems great -- now I feel stupid for my

if (path / 'StructureDataset-metadata.json').exists():
    ...
elif (path / 'cldf-metadata.json').exists():
    ...

(<_<)"

[xrotwang]

I feel stupid for not specifying something like "known locations" for the metadata files in the CLDF standard :)

[johenglisch]

Well, checking for the dc:conformsTo bit feels more reliable to me.