Rotary positional embeddings for conformer

[pzelasko]

Just throwing the idea around in case somebody wants to pick it up, I wanted to do it for some time but can't find the time. There's a new position embedding type called RoPE that yield improvements across different NLP tasks and now also ASR. Shouldn't be too difficult to add, and there is an implementation here: https://github.com/lucidrains/rotary-embedding-torch

Original paper: https://arxiv.org/abs/2104.09864
Application in ASR: https://arxiv.org/abs/2107.05907