Add an attention window, as a rectangular window on the attention scores. The aim is to help the network focus on local patterns, without affecting the attention scores themselves.
Add an attention window, as a rectangular window on the attention scores. The aim is to help the network focus on local patterns, without affecting the attention scores themselves.