The results in the paper can not be reproduced

[ZHuangyeei]

Are there any special settings or parameters that need to be modified? I didn't change it, but I couldn't reproduce the points in the paper. Also, the trained checkpoint on refer-kitti-v2 you provided is invalid. Can you upload it again?

[hszh]

I encountered the same problem, using the provided weights on the refer-kitti dataset only yields a HOTA value of 50.49, which is lower than the results in the paper by nearly 2 points.

Should we adjust any hyper parameters? @zyn213

[xzb666]

@HongShen-Z How long have you been training?

[hszh]

@HongShen-Z How long have you been training?

I use the provided weights in this repo: https://github.com/zyn213/TempRMOT/releases/download/v1.0/checkpoint_rk.pth

[Shuaicong97]

@hszh @ZHuangyeei I'm trying to evaluate it. But there are some parameters for which I don't know the paths. Could you please reply?
--GT_FOLDER /data/Dataset/refer-kitti/KITTI/training/image_02
Refer to #18
Thank you in advance!

[hszh]

@Shuaicong97 You can find the SEQMAP_FILE in here

[hszh]

@HongShen-Z How long have you been training?

@xzb666 Hi, I have tried 60 epochs and 100 epochs, the best hota I got is 49. How about you?

[xzb666]

The same. I'm over 48. How long do you train each time?

…

---Original--- From: ***@***.***> Date: Sun, Jan 5, 2025 13:22 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [zyn213/TempRMOT] The results in the paper can not be reproduced(Issue #16) @HongShen-Z How long have you been training? @xzb666 Hi, I have tried 60 epochs and 100 epochs, the best hota I got is 49. How about you? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[hszh]

One epoch? About 20 min on two 4090

[Shuaicong97]

@Shuaicong97 You can find the SEQMAP_FILE in here

Thanks. What about GT_FOLDER? I located the training set folder where there are no gt files. However after inference there is gt file under default/results_epoch0/<video_id>/white-cars-are-moving-in-the-direction-of-the-ego-car/gt.txt. Should GT_FOLDER be the same as TRACKERS_FOLDER?

[shd1998]

@HongShen-Z How long have you been training?

@xzb666 Hi, I have tried 60 epochs and 100 epochs, the best hota I got is 49. How about you?

@hszh @xzb666 Hi, I also achieved a best result of 49. Have you managed to get a score above 50 later on?