GLUE Baselines

This repo contains the code for baselines for the Generalized Language Understanding Evaluation (GLUE) benchmark. See our paper for more details about GLUE or the baselines.

Deprecation Warning

Use this code to reproduce our baselines. If you want code to use as a starting point for new development, though, we strongly recommend using jiant instead—it's a much more extensive and much better-documented toolkit built around the same goals.

Dependencies

Make sure you have installed the packages listed in environment.yml. When listed, specific particular package versions are required. If you use conda, you can create an environment from this package with the following command:

conda env create -f environment.yml

If you use apple silicon, you can create an environment from this package with the following command:

CONDA_SUBDIR=osx-64 conda env create -f environment.yml

Note: The version of AllenNLP available on pip may not be compatible with PyTorch 0.4, in which we recommend installing from source.

Activate environment

conda activate glue

If you use apple silicon, you can also lock the architecture to x86_64:

conda config --env --set subdir osx-64

Downloading WNLI

We provide a convenience python script for downloading all WNLI data and standard splits.

python download_glue_data.py --data_dir glue_data --tasks WNLI

Running WNLI

To run our baselines, use src/main.py.

python src/main.py --exp_dir experiments_glue --run_dir experiments_glue/wnli_real_run --train_tasks wnli --eval_tasks none --cuda -1 --word_embs_file embeddings/glove.840B.300d.txt

NB: The version of AllenNLP used has issues with tensorboard. You may need to substitute calls from tensorboard import SummaryWriter to from tensorboardX import SummaryWriter in your AllenNLP source files.

GloVe, CoVe, and ELMo

To use GloVe, download from this link and place in the embeddings folder: https://nlp.stanford.edu/data/glove.840B.300d.zip

We use the CoVe implementation provided here. To use CoVe, clone the repo and fill in PATH_TO_COVE in src/models.py and set --cove to 1.

We use the ELMo implementation provided by AllenNLP. To use ELMo, set --elmo to 1. To use ELMo without GloVe, additionally set --elmo_no_glove to 1.

Reference

If you use this code or GLUE, please consider citing us.

 @unpublished{wang2018glue
     title={{GLUE}: A Multi-Task Benchmark and Analysis Platform for
             Natural Language Understanding}
     author={Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill,
             Felix and Levy, Omer and Bowman, Samuel R.}
     note={arXiv preprint 1804.07461}
     year={2018}
 }

Feel free to contact alexwang at nyu.edu with any questions or comments.