Calculation of BLEU score

[sachin-vashistha]

Hi Bsantraigi
You have provided great tutorials. I saw your transliteration.ipynb.
In this, in Input cell "In[37]", you have put the line 'pbar.set_description(f"Epoch: {_e} >> Loss: {batch_loss/(bxi+1):2.2F}:")', but in the output, there is no information about Loss.
Also, can you explain how you calculated the BLEU score? I guess for every Epoch, you are calculating the BLEU score 4 times. I didn't understand your "small_test()" function. Your final BLEU score is 0.6288471674183466. It is a very high BLEU score. Isn't it represent that model is overfitting?

[bsantraigi]

Hi @sachin-vashistha
Good to know that you are finding these tutorials useful. Here are the answers to your query

1. I had used tqdm to show progressbar and update the description dynamically, pbar.set_description(f"Epoch: {_e} >> Loss: {batch_loss/(bxi+1):2.2F}:") . This gets rendered only when the code is executed. You can use the colab link in the readme (https://github.com/bsantraigi/Tensorflow-RNN-Tutorials#transliteration) to execute it on cloud gpu and print the loss. Or you can change it to a print statement. print(f"Epoch: {_e} >> Loss: {batch_loss/(bxi+1):2.2F}:")
2. Yes the bleu scores are quite high. But since it is calculated on validation set, it shows the model is training well and not overfitting. The reason for why it's high is that it's a relatively simpler problem (than other seq2seq tasks like NMT) and only sequences of smaller lengths have been considered. See in cell In [5], I have set MAX_SEQ_LEN = 20.