error

[xjyisok]

During training the first time after validation and save a checkpoint and then continue to train,it will output error below

thanks for answering

[Hatins]

Did you set wandb offline? This error may be caused by setting the wandb offline or did not set the configs about the wandb completely.

[xjyisok]

My wandb is online. The first 20,000 steps of the training are normal, but this error is reported after validation and checkpoint saved.it seems that there has something wrong when restarting training after val.
This is my wandb config

and the wandb is infact online

[Hatins]

That's strange since I only meet this problem when I set the wandb as offline, and I have run this code successfully for the recent rebuttal. I will address this problem, but it may depend on the final result about the acceptance of our work in the next week.

[Barry-30]

I also encountered this problem. I rolled back wandb to the same version as RVT, 0.14.0, and it solved it.

[Hatins]

Thanks, I think it might be pip bumping up the default wandb version that caused the issue.

[Barry-30]

But I ran into problems when I tried to use multiple gpus to train the task.

[Hatins]

The setting of the batch_size, number_work, and the number of GPUs may cause this. What's your setting?

[Hatins]

You can try to REDUCE the batch_size and number_worker and run again. But I recommend using the same setting as I suggested to get the same results.

[Barry-30]

I use 4 A6000(48G)
BATCH_SIZE_PER_GPU=4
TRAIN_WORKERS_PER_GPU=6
EVAL_WORKERS_PER_GPU=2
But no matter how I adjust the workers I have problems.

[Barry-30]

Yes, I have no problem training with a single GPU, but I want to train faster.

[Hatins]

This is too big since the number of the test datasets is not so big. so you can try to reduce the batch size and number work off of the test. Then you can train faster.

[Barry-30]

Thank you for your reminder. It is true that I neglected the test size. After I set the batch_size of eval separately, I can train normally.

[Hatins]

You're welcome, plz note to adjust the learning_rate when you change the batch size, according to the relation mention in the RVT.

[Barry-30]

Yes, I followed previous work and used the following scaling formula.