training issues

[asmodaay]

Hi, i tried to train model with only LJ data, and with only own data, with fp16 and with fr32, with 1 gpu and with 3 gpu, but everywhere i have this

Always los is Nan.
When i start with pretrained chekpoint your code return this:

I solve it by changing def load_checkpoint , but loss is nan(

do u have any ideas what am i doing wrong?

[wietsedv]

It does not work properly with the latest PyTorch version (I have not searched for exact problems yet). PyTorch 1.0.0 works for me. I have some edge case problems with 1.0.1 so I would install 1.0.0.

[asmodaay]

ty, i will try!

[rafaelvalle]

replace .byte() with .bool()
https://github.com/NVIDIA/flowtron/blob/master/flowtron.py#L33

[asmodaay]

replace .byte() with .bool()
https://github.com/NVIDIA/flowtron/blob/master/flowtron.py#L33

This helps me, and i got this attention plot, but inference samples on this speaker doesnt have any meaningful speech. Is attention ok?

[rafaelvalle]

This attention plot looks really good!
Can you show both attention layers? This is only layer 1.

[asmodaay]

This attention plot looks really good!
Can you show both attention layers? This is only layer 1.

I have ok quality on LJ speaker and shit quality on my own speaker(~60 min data)

[rafaelvalle]

Both alignment plots look good.
Can you perform inference on your speaker and share the alignment plots and audio?

[asmodaay]

samples.zip
Here is 2 sample with lj speaker and another
1st plot lj
2nd not lj

[rafaelvalle]

Looks like the model hasn't learned to attend on your speaker.

[asmodaay]

Looks like the model hasn't learned to attend on your speaker.

Do u have any ideas how to fix it? more data or smth like that, ill try to add my speaker in libri group and finetune lj checkpoint

[daun-io]

I'm currently training my own data (~2hrs) from LibriTTS pretrained model and looks like the model hasn't learned to attend yet too.

This is my configuration.

{
    "train_config": {
        "output_directory": "/some_output_directory/",
        "epochs": 10000000,
        "learning_rate": 1e-4,
        "weight_decay": 1e-6,
        "sigma": 1.0,
        "iters_per_checkpoint": 5000,
        "batch_size": 4,
        "seed": 1234,
        "checkpoint_path": "/some_checkpoint_path/",
	    "ignore_layers": [],
        "include_layers": ["speaker", "encoder", "embedding"],
        "warmstart_checkpoint_path": "",
        "with_tensorboard": true,
        "fp16_run": true
    },
    "data_config": {
        "training_files": "../../path_to_datasets/train_filelist.txt", 
        "validation_files": "../../path_to_datasets/val_filelist.txt",
        "text_cleaners": ["flowtron_cleaners"],
        "p_arpabet": 0.5,
        "cmudict_path": "data/cmudict_dictionary",
        "sampling_rate": 22050,
        "filter_length": 1024,
        "hop_length": 256,
        "win_length": 1024,
        "mel_fmin": 0.0,
        "mel_fmax": 8000.0,
        "max_wav_value": 32768.0
    },
    "dist_config": {
        "dist_backend": "nccl",
        "dist_url": "tcp://localhost:54321"
    },
    
    "model_config": {
        "n_speakers": 1,
        "n_speaker_dim": 128,
        "n_text": 185,
        "n_text_dim": 512,
        "n_flows": 2,
        "n_mel_channels": 80,
        "n_attn_channels": 640,
        "n_hidden": 1024,
        "n_lstm_layers": 2,
        "mel_encoder_n_hidden": 512,
        "n_components": 0,
        "mean_scale": 0.0,
        "fixed_gaussian": true,
        "dummy_speaker_embedding": false,
        "use_gate_layer": true
    } 
}

[rafaelvalle]

Rule of thumb is to train the model until it starts overfitting or the loss plateaus.

[daun-io]

Yeah I have trained several tacotron 2 networks(nvidia implementation) before with pretrained models and my own datasets. I tend to see the trend that since my dataset is relatively small (~1hours or ~2hours) the validation loss shows overfitting in very early stage of training (like 1k steps). However the results generated by the model tends to get better even after overfitting is started and the alignments looked fine.
In this training I found that flowtron model also overfits really fast (like in several thousand steps) and in that stage of training the model couldn't really reconstruct the features of my datasets. When I test the overfitted model (trained more than 100k steps) it shows poor results on alignment for new sentences. Is there any empirical way to figure this out?

[rafaelvalle]

Combine your speaker with data from other speakers and train again.
Can you share inference mel-spectrograms and alignment plots for your speaker with your fine-tuned model?

[daun-io]

I've currently combined my speaker data with libritts-100 and training it from scratch without pretrained model with batch size of 1 and I haven't seen any sign of alignments from the results yet.

Do you know the empirical steps or epochs needed to see the alignments from tensorboard?
And here's my previous model's (the one adaptated from the pretrained model) mel-spectrograms and alignment plots.

You can also hear it, The sentence is "It is well know that deep generative models have a deep latent space!"
sid0_sigma0.5.wav.zip