Transformer Language Models Can Represent n-gram Language Models

Code accompanying the NAACL 2024 publication "Transformer Language Models Can Represent n-gram Language Models".

Getting started with the code Clone the repository:

$ git clone git@github.com:rycolab/transformer-ngrams.git
$ cd transformer-ngrams

It may be beneficial to create a new Python virtual environment (e.g., using conda). We aim at Python 3.10 version and above.

You can then install the package in editable mode:

$ pip install -e .

We use pytest to unit test the code. You can run the tests with:

$ pytest tests/