ReproHum NeuralREG reproduction

This repository is a clone and modified version of the repo: https://github.com/ThiagoCF05/NeuralREG to reproduce the human evaluations that was conducted by the original author(s).

NeuralREG

This project provides the data and models described on the ACL 2018 paper "NeuralREG: An end-to-end approach to referring expression generation" (available here).

NeuralREG models

NeuralREG+Seq2Seq

Seq2Seq version. To train and evaluate the model, you may update the variable paths in the script and run the following command:

python seq2seq.py --dynet-gpu

NeuralREG+CAtt

Concatenative attention version. To train and evaluate the model, you may update the variable paths in the script and run the following command:

python3 attention.py --dynet-gpu

NeuralREG+HierAtt

Hierarcical attention version. To train and evaluate the model, you may update the variable paths in the script and run the following command:

python hierattention.py --dynet-gpu

Data

WebNLG

The original and delexicalized versions of the WebNLG corpus used in our experiments.

Referring Expressions

Training, development and test referring expressions sets and vocabularies. This is the official data used to train and evaluate the models. It was extracted from WebNLG/ using the command:

python preprocessing.py [IN_PATH] [OUT_PATH] [STANFORD_PATH]

Baselines

OnlyNames

OnlyNames baseline. The model may be executed by the following command:

python2.7 only_names.py

Castro Ferreira et al.

This baseline is an adaptation of the model described in this paper. The model may be executed by the following commands:

python2.7 reg_train.py
python2.7 reg_main.py

Evaluation

eval/

Automatic evaluation scripts to extract information about the referring expression collection (corpus.py), to obtain the results depicted in the paper (evaluation.py) and to test statistical significance (statistics.R)

humaneval/

Human evaluation scripts to obtain results depicted in the paper (stats.py) and to test statistical significance (statistics.R)

Citation

@InProceedings{ferreiraetal2018b,
  author = 	"Castro Ferreira, Thiago
		and Moussallem, Diego
		and K{\'a}d{\'a}r, {\'A}kos
		and Wubben, Sander
		and Krahmer, Emiel",
  title = 	"NeuralREG: An end-to-end approach to referring expression generation",
  booktitle = 	"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
  year = 	"2018",
  publisher = 	"Association for Computational Linguistics",
  pages = 	"1959--1969",
  location = 	"Melbourne, Australia",
  url = 	"http://aclweb.org/anthology/P18-1182"
}

Author: Thiago Castro Ferreira

Date: 15/12/2017 (Updated on June 3rd 2019)