The results of the tartanair assessment I trained myself on were not good

[keerheer]

Hello, thank you very much for your work.
When I reproduced your work, I trained on tartanAir using the commands you gave me: I ran the following command: python train.py--steps=240000--lr=0.00008--name=. After completing the training, I conducted an evaluation on tartanair, saving a checkpoint every 10,000 steps. I used the last checkpoint (checkpoint_240000) for the evaluation and also downloaded the dpvo.pth you provided for further evaluation. I used your recommended command: python evaluate_tartan.py--trials=5--split=validation--plot--save_trajectory, which yielded the following results:

AUC":0.8542202128818401,"AVG":0.13653399537956934（your）
"AUC":0.7582382139923,"AVG":0.27449909769417025(ours)

The results on the left are from your dpvo.pth, and those on the right are from my checkpoint_240000.pth. The results show a significant difference. I would like to know why this is happening, as I haven't made any changes to the other configurations of the training code. I am using a single A5000 with 24G of video memory, and the training took about 4.2 days. I look forward to your response.

[zachteed]

Hi, yes those results look quite a bit worse. Was the validation accuracy higher for earlier checkpoints (the training script should compute validation accuracy during tarining)? I've noticed the validation accuracy fluctuates quite a bit during training and we take the model with the best validation accuracy.