Training Instability and Performance Issues on RGB-stacking with Reduced Hardware Constraints

[kamadalao]

Thank you for your time and for this great project!

I am attempting to train/fine-tune CoTracker3Online on the RGB-stacking dataset (and Kubric) using a single node with 8x A100 (40GB). However, I am experiencing significant training fluctuations and the final tracking performance is suboptimal compared to the expected results.

I am using the following command:
python train_on_kubric.py --batch_size 1 --num_steps 200000 \ --ckpt_path ./ --model_name cotracker_three --save_freq 200 --sequence_len 32 \ --eval_datasets tapvid_davis_first tapvid_stacking --traj_per_sample 384 \ --sliding_window_len 16 --train_datasets kubric --save_every_n_epoch 1 \ --evaluate_every_n_epoch 1 --model_stride 4 --dataset_root ${path_to_your_dataset} \ --num_nodes 1 --num_virtual_tracks 64 --mixed_precision --corr_radius 3 \ --wdecay 0.0005 --linear_layer_for_vis_conf --validate_at_start --add_huber_loss