reduction window is vital for the model to pick up alignment.

[bfs18]

The hparams.py says n_frames_per_step=1, # currently only 1 is supported, but reduction window is very important for them model to pick up alignment. Using a reduction window can be considered as dropping teacher forcing frames at equal intervals, and thus increases the information gap between the teacher forcing input and the target. Tacotron2 tends to predict the target from the autoregressive input (teacher forcing input at training) without exploiting the conditional text if the information gap is not large enough.
The reduction window can be replaced by a frame dropout trick if it is not continent to implement in the current code. Just set the teacher forcing input frames to the global mean according to a certain percentage.
In implement this in my fork. It can pick up alignment at much earlier steps without warmstart.
my fork

NVIDIA-tacotorn2

[xDuck]

Read up a bit on your implementation and it seems very promising. Going to give it a go with a fork I've been working on that is struggling to learn attention fully. I was looking into applying something similar (but not nearly as elegant) myself.

Can you provide a link to the paper you reference in your fork's README?

[bfs18]

Hi, the paper is available at https://arxiv.org/abs/1909.01145

[onyedikilo]

@bfs18 Hi, tried your fork but somehow I am getting NaN's on gradient.norm and mi loss, any ideas? I trained master successfully with the same data.

[bfs18]

@bfs18 Hi, tried your fork but somehow I am getting NaN's on gradient.norm and mi loss, any ideas? I trained master successfully with the same data.

Hi @onyedikilo

1. You can only use drop frame first, just set use_mmi=False.
2. If you would like to use mmi, make sure that the blank index and vocab size are correctly setted (torch.CTCLoss doc). Besides silent symbols, such as 'SPACE' and punctuation, should be avoided in the CTC target. Finally you may try reducing gaf to 0.1.

[onyedikilo]

@bfs18 Sorry I couldn't understand what you meant with

You can only use drop frame first

Can you explain it in different words?

[bfs18]

Hi @onyedikilo
I add several new options in hparams.py in my fork.

• use_mmi (use mmi training objective or not)
• use_gaf (use gradient adaptive factor or not, to keep the max norm of gradients from the taco_loss and mi_loss approximately equal)
• max_gaf (maximum value of gradient adaptive factor)
• drop_frame_rate (drop teacher-frocing input frames to a certain rate)

When setting use_mmi=False and drop_frame_rate to a value in range (0., 1.), only drop frame trick is used.

[hadaev8]

@bfs18
Why you have this line?
https://github.com/bfs18/tacotron2/blob/master/train.py#L253

[bfs18]

@bfs18
Why you have this line?
https://github.com/bfs18/tacotron2/blob/master/train.py#L253

Hi @hadaev8 , this line has no influence on the numerical values of gradients. When calculating taco_loss, the variables of CTC recognizer is not used, so the gradients of taco_loss with respect to these variables are None. The gradients become zero tensors after adding this line.

[onyedikilo]

I can confirm that the alignment picks up significantly faster with my data set.

[bfs18]

I can confirm that the alignment picks up significantly faster with my data set.

Hi @onyedikilo , thanks a lot for your confirmation.

[hadaev8]

@bfs18
Any ideas why my alignment looks like this with CTC loss?
https://i.imgur.com/17Wz22v.png

[bfs18]

@bfs18
Any ideas why my alignment looks like this with CTC loss?
https://i.imgur.com/17Wz22v.png

Hi @hadaev8 , This is caused by that the CTC loss is over weighted. When CTC loss is over weighted, the model would depend more on the text input to reduce the total loss. It leads to a diagonal alignment combined with the Local Sensitive Attention.

Setting hparams.use_gaf=True and a smaller hparams.max_gaf, such as 0.1, would solve the problem.

[hadaev8]

Well, I read again paper
They say

In Tacotron2, the attention context is concatenated to the LSTM out-put and projected by a linear transform to predict the Mel spectrum.This means the predicated Mel spectrum contains linear componentsof the text information. If we use this Mel spectrum as the input tothe CTC recognizer, the text information is too easily accessible forthe recognizer. This may cause the text information to be encodedin a pathological way in the Mel spectrum and lead to a strict di-agonal alignment map (one acoustic frame output for one phonemeinput) combined with location-sensitive attention. So before the lin-ear transform operation, we add an extra LSTM layer to mix the textinformation and acoustic information.

Should you point where should be this lstm layer?

[bfs18]

Well, I read again paper
They say

In Tacotron2, the attention context is concatenated to the LSTM out-put and projected by a linear transform to predict the Mel spectrum.This means the predicated Mel spectrum contains linear componentsof the text information. If we use this Mel spectrum as the input tothe CTC recognizer, the text information is too easily accessible forthe recognizer. This may cause the text information to be encodedin a pathological way in the Mel spectrum and lead to a strict di-agonal alignment map (one acoustic frame output for one phonemeinput) combined with location-sensitive attention. So before the lin-ear transform operation, we add an extra LSTM layer to mix the textinformation and acoustic information.

Should you point where should be this lstm layer?

Hi, the paper uses a internal Tensorflow implementation. It is a bit different from the open-sourced fork. In the open-sourced fork a ff_layer with relu activation is used to mix the information. It is this line https://github.com/bfs18/tacotron2/blob/8f8605ee0f67f6f571e74725030f16b13e4c7d2d/model.py#L388

[xDuck]

Finally got around to trying out your fork on my modified spectrums and I can confirm it picked up attention much faster! Thanks!