Reason for repeated output tokens

[Mo-Saif]

Hi
I think the reason your model does not learn to end sentences properly is that you are using a greedy decoder. I think if you used beam search instead for decoding, you will have far better results.
You can learn more from this video
https://www.youtube.com/watch?v=RLWuzLLSIgw

[jumerckx]

Hey,
Thanks for the feedback and sorry for the late reply, the notification must have slipped past me.
A beam search decoder should indeed increase accuracy, but I'm sure there's a more fundamental flaw at play here.
Similar models (e.g. The pytorch implementation I used as a reference) prove that a simple seq2seq model with greedy decoding should be able to translate at a level that is at least "ok". (My model however seems to be unable to learn this)
I'm trying to keep the model as bare-bones as possible as a means for me to understand the model and to convey the essence better.
I'd love to hear if you have any more remarks or potential improvements.
Thanks
Jules

[Mo-Saif]

I see. I am implementing "Pointer Networks" paper currently in Julia. The paper is based on the seq2seq architecture. I couldn't get it to work yet. If I find my problem relevant to your case I will write a comment here.

[kadir-gunel]

Hello, I think your model does not give good results because you set the learning rate too high. Could you try sth like 1e-3 or 1e-4. And 70 epochs is too much.

[jumerckx]

Thanks for taking a look @kadir-gunel!
It has been a long time since I've looked into this, however I'm very keen on making this work.
Flux's api has changed (is changing) a lot but when I find the time, I'll try to make the needed changes to make the code run again.
I believe I've already tried lowering the learning rate but my memory is fuzzy so I could be wrong.

If you, by any chance, look further into my code, be sure to let me know of any problems!