Assertion Error for En-De Translation using latent search and pretrained model .

[spprabhu]

Hi Raphael I am getting following results on WMT En-De Translation for

1. Fast Translation
   2.One REfinement step

python run.py --opt_dtok wmt14_ende --use_pretrain --test --evaluate
[OPTS] Model tag: dtok-wmt14_ende
Running on 1 GPUs
loading pretrained model in ./mydata/shu_trained_wmt14_ende.pt
translating: 100.0%
Average decoding time: 15ms, std: 1
BLEU = 22.304035169763978

python run.py --opt_dtok wmt14_ende --use_pretrain --opt_Trefine_steps 1 --test --evaluate
[OPTS] Model tag: dtok-wmt14_ende
Running on 1 GPUs
loading pretrained model in ./mydata/shu_trained_wmt14_ende.pt
translating: 100.0%
Average decoding time: 38ms, std: 1
BLEU = 24.147135058514433

They are inline with the specified results.However I am not able to implement the model using Latent Search as it is giving Assertion Error.

python run.py --opt_dtok wmt14_ende --use_pretrain --opt_Trefine_steps 1 --opt_Tlatent_search --opt_Tteacher_rescore --test --evaluate
[OPTS] Model tag: dtok-wmt14_ende
Running on 1 GPUs
Traceback (most recent call last):
File "run.py", line 207, in
assert os.path.exists(pretrained_autoregressive_path)
AssertionError

[zomux]

You need to download my pretrained autoregressive models. Please check the Github page, there are the command for downloading them.

[zomux]

BTW, your average decoding times are 15ms and 38ms, faster than my results. Amazing.

[spprabhu]

Hi This are the speeds for Latent Search Version.

python run.py --opt_dtok wmt14_ende --use_pretrain --opt_Trefine_steps 1 --opt_Tlatent_search --opt_Tteacher_rescore --test --evaluate
[OPTS] Model tag: dtok-wmt14_ende
Running on 1 GPUs
loading pretrained model in ./mydata/shu_trained_wmt14_ende.pt
translating: 100.0%
Average decoding time: 205ms, std: 89
BLEU = 25.140153770802392

Just for Minor increase in BLEU it takes a lot of time for Decoding.

[spprabhu]

Any Reasons for such Huge gap in Specified vs Real Results.

[zomux]

Did you try a second time?

[spprabhu]

ALthough I am using T4 GPUs

[zomux]

Can you report the results with

python run.py --opt_dtok wmt14_ende --use_pretrain --opt_Trefine_steps 1 --opt_Tlatent_search --test --evaluate

[spprabhu]

Sure.

[zomux]

@spprabhu Gonna sleep, let me check the preprocessing part tomorrow (JST TIMEZONE)

[spprabhu]

Ok.

[spprabhu]

Hi These are the results with Latent search without the Teacher Rescoring

python run.py --opt_dtok wmt14_ende --use_pretrain --opt_Trefine_steps 1 --opt_Tlatent_search --test --evaluate
[OPTS] Model tag: dtok-wmt14_ende
Running on 1 GPUs
loading pretrained model in ./mydata/shu_trained_wmt14_ende.pt
translating: 100.0%
Average decoding time: 157ms, std: 67
BLEU = 25.005498749025612

[spprabhu]

I guess there isn,t much improvement using Latent Search and Teacher Rescoring.I need to know what models are you using for your autoregressive methods though.

If we can compare time by using Different Autoregressive Models ( For eg: Transformer,GPT-2,Fairseq )and Benchmark it.

[spprabhu]

Hi Raphael Any Updates.

[zomux]

Hi, I submitted a job for evaluation the decoding again. But I did that many times, I can't be over 100ms. So I guess your GPU may have a low performance, so that multiple latent variables can't be computed simultaneously. This is shown by the result without the Teacher Rescoring.

[zomux]

My autoregressive baseline model is just a normal transformer.