The problem that the model cannot converge

[white1-doggy]

Hi, I have some training questions to ask.

When I train the AO model at 2 A100 80G, Batch_size=8, and train the lrs3 and lrs2 data sets simultaneously, errors will be reported out of memory and the training time will reach an epoch of 5 hours.

When only the ls2 data set is used, Batch_size=16, num_workers=4, an error will also be reported out of memory. In this case, the epoch is 20 minutes.

When only ls2 data set is used, Batch_size=8, num_workers=4, the epoch is 40 minutes, but loss=nan occurs when multiple epochs are used.

All the above problems have caused me to be unable to train normally. Could you please tell me the time and details of your training of this model in detail? Thank you very much！！

[jeonhuhuhu]

Hi, I have some training questions to ask.

When I train the AO model at 2 A100 80G, Batch_size=8, and train the lrs3 and lrs2 data sets simultaneously, errors will be reported out of memory and the training time will reach an epoch of 5 hours.

When only the ls2 data set is used, Batch_size=16, num_workers=4, an error will also be reported out of memory. In this case, the epoch is 20 minutes.

When only ls2 data set is used, Batch_size=8, num_workers=4, the epoch is 40 minutes, but loss=nan occurs when multiple epochs are used.

All the above problems have caused me to be unable to train normally. Could you please tell me the time and details of your training of this model in detail? Thank you very much！！

Have you looked at link #7 or #14 ?

If you refer to the link, I think it will be a good hint for solving the Loss=nan problem.

And out of curiosity, how did you solve problem #16?

I can't download or create data. Can you share a solution?