CLDF-Meta: A catalogue of CLDF datasets

How to cite

If you use these data please cite this dataset using the DOI of the particular released version you were using

Description

This dataset is licensed under a CC-BY-4.0 license

Available online at meta.clld.org

Basic Workflow

Creating the meta database is a three-step process:

1. Download metadata for existing datasets from Zenodo. This will update the metadata in raw/zenodo-metadata.json.

   $ cldfbench cldf-meta.updatemd cldfbench_cldf_meta.py

2. Download the datasets themselves. They will be downloaded into the raw/datasets/ folder.

   $ cldfbench download cldfbench_cldf_meta.py

3. Look through the datasets and create the meta database. This will update the CLDF dataset in cldf/ and also add files that don't contain any CLDF data to etc/not-cldf.csv, so they can be avoided in the future.

   $ cldfbench makecldf cldfbench_cldf_meta.py

Important files

• raw/zenodo-metadata.json: contains the metadata downloaded from Zenodo. This file is updated automatically by the updatemd command.
• etc/blacklist.csv: contains DOIs for datasets that should be excluded from the meta database (e.g. the CLDF version of Glottolog). This file is meant to be edited manually.
• etc/whitelist.csv: contains DOIs for datasets that should explicitly be added to the meta database. This file meant to be edited manually.
• etc/not-cldf.csv: contains a list of dataset files that are known to not contain CLDF. These files will not be downloaded or scanned for CLDF data. This file is updated automatically by the makecldf command.

Using a Personal Access Token to access Zenodo

Since this project involves downloading a lot of data, there is a non-zero chance that the updatemd or download commands might hit the rate limits for Zenodo's API.

If you need to extend the rate limit, you can set up a Personal Access Token and add it to the $CLDF_META_ACCESS_TOKEN environment variable before running cldfbench:

$ export CLDF_META_ACCESS_TOKEN=AbCdEfG[…]
$ cldfbench download cldfbench_cldf_meta.py

Update 2026-01-14: Recently Zenodo limited the number of search results per request from 100 to 25, which means updatemd will make four times as many requests to the Zenodo API to fetch the same data. Therefore using an access token is highly recommended to reduce the number of requests to the server and reduce the likelihood of hitting the rate limit.

CLDF Datasets

The following CLDF datasets are available in cldf:

• CLDF Generic at cldf/cldf-metadata.json