Pre-trained WikiText-103 language model

[mrzpx]

where do i get this pre-trained model?

[wulouzhu]

Hello, do you know how to download the lang-8-en-1.0.zip data?

[wulouzhu]

have you found it?

[mrzpx]

https://sites.google.com/site/naistlang8corpora/

[wulouzhu]

Oh, I have found the lang-8 data myself. I means Pre-trained WikiText-103 language model .Thank you!

[wulouzhu]

@mrzpx
Hi, I have downloaded the Pre-trained WikiText-103 language model here https://github.com/pytorch/fairseq/blob/master/examples/language_model/README.md. Then I use it for testing model. But I met the error:
Missing key(s) in state_dict: "decoder.embed_tokens.weight", "decoder.adaptive_softmax.head.weight".
Unexpected key(s) in state_dict: "decoder.embed_tokens._float_tensor", "decoder.embed_tokens.embeddings.0.0.weight", "decoder.embed_tokens.embeddings.0.1.weight", "decoder.embed_tokens.embeddings.1.0.weight", "decoder.embed_tokens.embeddings.1.1.weight", "decoder.embed_tokens.embeddings.2.0.weight", "decoder.embed_tokens.embeddings.2.1.weight", "decoder.adaptive_softmax.head._float_tensor", "decoder.adaptive_softmax.head.word_proj.weight", "decoder.adaptive_softmax.head.class_proj.weight".
size mismatch for decoder.adaptive_softmax.tail.0.0.weight: copying a param of torch.Size([256, 1024]) from checkpoint, where the shape is torch.Size([1024, 256]) in current model.
size mismatch for decoder.adaptive_softmax.tail.1.0.weight: copying a param of torch.Size([64, 1024]) from checkpoint, where the shape is torch.Size([1024, 64]) in current model.

Did you meet the error?

[wulouzhu]

@mrzpx @rgcottrell
Hello, how should I pre-train WikiText-103 language model. Hope to receive your reply.

[young-zonglin]

Maybe you can try to train 5-gram language model with kenlm and en-wiki corpus.

[wulouzhu]

@young-zonglin
Did you do it in this way? And did it work?

[rgcottrell]

I just used the pre-trained model from the Fairseq repo. It's possible that it's been updated for PyTorch 1.0 and new model distribution formats and no longer functional for the older version used when the project was started.

[rgcottrell]

Here's the link to the wiki model from the GitHub repo history, I think:

https://dl.fbaipublicfiles.com/fairseq/models/wiki103_fconv_lm.tar.bz2

[wulouzhu]

@rgcottrell
Thanks for your reply. There is an another problem.
When I ran the command as follow:
python3 ./generate.py
../data-bin/lang-8-fairseq
--path ../checkpoints/lang-8-fairseq-cnn/checkpoint_best.pt
--batch-size 128
--beam 5
--nbest 1
Something error happened:
args: {'path': None, 'data': None, 'task': 'language_modeling', 'raw_text': True, 'sample_break_mode': 'eos', 'output_dictionary_size': -1, 'self_target': False, 'future_target': False, 'past_target': False, 'num_shards': 1, 'shard_id': 0, 'max_tokens': None, 'max_sentences': None, 'tokens_per_sample': 1024}
args.data: None
Traceback (most recent call last):
File "./generate.py", line 237, in
main(args)
File "./generate.py", line 94, in main
fluency_scorer = FluencyScorer(args.lang_model_path, args.lang_model_data)
File "/home/nlp/WJF/pytorch-human-performance-gec-master/fairseq-scripts/fluency_scorer.py", line 58, in init
self.task = tasks.setup_task(self.args)
File "/home/nlp/WJF/pytorch-human-performance-gec-master/fairseq/fairseq/tasks/init.py", line 19, in setup_task
return TASK_REGISTRY[args.task].setup_task(args)
File "/home/nlp/WJF/pytorch-human-performance-gec-master/fairseq/fairseq/tasks/language_modeling.py", line 92, in setup_task
dictionary = Dictionary.load(os.path.join(args.data, 'dict.txt'))
File "/usr/lib64/python3.6/posixpath.py", line 80, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

It seems that I should provide the language model dict path.
But when I replaced "./generate.py " with "../fairseq/generate.py", it worked.
Should I provide the language model dict path or replace "./generate.py " with "../fairseq/generate.py"

[rgcottrell]

The custom ./generate.py script is used to add the language model and fluency scorer to the output, so you will want to use it to get the best version of the results. If you look at some of the other batch files, you will see some things like:

generate-lang8-cnn-rawtext.bat

python .\generate.py^
    ..\test\lang-8^
    --path ..\checkpoints\lang-8-fairseq-cnn\checkpoint_best.pt^
    --batch-size 128^
    --beam 5^
    --nbest 12^
    --lang-model-data ..\data-bin\wiki103^
    --lang-model-path ..\data-bin\wiki103\wiki103.pt^
    --raw-text^
    --source-lang en^
    --target-lang gec

So once you have the wiki language model installed, try adding the --lang-model-data and lang-model-path options and see if that works.

Sorry but it's been a while since I ran this code and I'm having some trouble remember exactly how things worked.

[wulouzhu]

@rgcottrell
So nice of you. I added the --lang-model-data and lang-model-path options and that worked.I could get translations. But when it calculated gleu score, something error happened in the gleu.py.

Traceback (most recent call last):
File "./generate.py", line 237, in
main(args)
File "./generate.py", line 219, in main
gleu_score = [g for g in gleu_scores][0][0] * 100;
File "./generate.py", line 219, in
gleu_score = [g for g in gleu_scores][0][0] * 100;
File "/home/nlp/WJF/pytorch-human-performance-gec-master/fairseq-scripts/gleu.py", line 199, in run_iterations
this_stats = [s for s in self.gleu_stats(i, r_ind=ref)]
File "/home/nlp/WJF/pytorch-human-performance-gec-master/fairseq-scripts/gleu.py", line 199, in
this_stats = [s for s in self.gleu_stats(i, r_ind=ref)]
File "/home/nlp/WJF/pytorch-human-performance-gec-master/fairseq-scripts/gleu.py", line 122, in gleu_stats
rlen = self.rlens[i][r_ind]
IndexError: list index out of range

Could you please tell me how to solved it. Thanks a lot

[rgcottrell]

I'm not sure what the issue is , but it could be that we didn't fix generate.py after finishing the GLEU score. You could try just commenting out the calls here. The GLEU score is only used to benchmark results against other papers and isn't part of the core algorithm.

You could also take a look at interactive.py, which was the final result of the project and what we used to show off the results with both a web-based and command line tool. It's a little smarter about how GLEU score processing is invoked.

[1228589545]

@wulouzhu
I also had an array overstep problem. How did you solve it?