Multi-head self/ cross attention

[charleneolive]

Hello:)
Thank you so much for sharing your code. It has been very useful in understanding the paper.

There is still something I don't quite get from the paper and the code. It From my understanding, the paper mentioned that the attention is calculated separately on multiple heads before being combined. However, I could not find how many heads are used in the code (not sure if I am confused about multi-head part). Could you point to where you defined the number of heads? I understand that in the paper, they did an experiment to analyse the number of attention heads for MHSA from 1 to 8.

Thank you.
Charlene

[Shannia-Shan]

Hi! Charlene,

Do you meet the problem :
\nTraceback (most recent call last):\n File "train.py", line 12, in \n from torch.cuda import amp\nImportError: cannot import name 'amp' from 'torch.cuda' (/usr/local/lib/python3.7/dist-packages/torch/cuda/init.py)\n

Thanks!

[charleneolive]

Nope I did not meet with any such error. What torch version are you using?

[Shannia-Shan]

Nope I did not meet with any such error. What torch version are you using?

Thanks!

I update the torch version as the following:
pip install torch==1.9.0+cu102 torchvision==0.10.0+cu102 torchaudio===0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

It is working now.

[HXLH50K]

Sorry about that，I‘m not implemented the multi-head part. I built the network by referring to the pictures in the paper. And I read the paper many times but not get the detail of multi-head.