Evaluating regressed pose

[tzamos]

hi,

I was experimenting with few modifications of your method, and I am currently trying to understand how to properly evaluate the regressed pose. In particular, say we are in forward_emm func in mp3d_loftr/src/loftr/loftr_module/transformer.py, and we have pred_reg_6d that contains the pose, with the first three elements of each batch being the translation and the next 6 is the 6d representation of the rotation.

consider I also have access to the point correspondences and everything that we need to compute inliers, then do we first need to unnormalize?

pred_reg_6d_r_norm = pred_reg_6d[...,3:].cpu().detach() * pose_std_6d[...,3:].cpu() + pose_mean_6d[...,3:].cpu()
pred_reg_t_norm = pred_reg_t * pose_std_6d[...,:3].cpu() + pose_mean_6d[...,:3].cpu()

I am getting very low inlier ratios with that pose (<0.1% inliers) while the estimated pose using loftr might have >70-80% inliers.

It is unclear to me, based on the code, how to transform the regressed pose so that it is in the original space and I can count inliers. Could you look into that?

Thanks!

[crockwell]

Hi,

Thanks for your interest. While I'm pretty busy and can't help debug, to me this sounds like you are on the right track since LoFTR is yielding many inliers. First of all, if you're using transformer output alone it won't be correct, since final output is a weighted combination of LoFTR output plus transformer output. If you are indeed using final output and it just seems to be in the wrong convention, you can look into a function to transform it into a more traditional rotation matrix here.

Best of luck!
Chris

[tzamos]

Thanks for your reply! In case you are not that busy now (assuming it was a paper deadline), I would like to be more specific about my issue. I was trying to check how good is the T_t, that is the regressed pose of the transformer. I am running the evaluation on the mp3d dataset. To get the results of T_t, as you report in the paper, I just replaced the combination of the pose
pred_T = pred_RT_wt[...,0] * pred_reg_t + (1-pred_RT_wt[...,0]) * loftr_pred_t
pred_R = pred_RT_wt[...,1] * pred_reg_6d[...,3:] + (1-pred_RT_wt[...,1]) * loftr_pred_R
by
pred_T = pred_reg_t
pred_R = pred_reg_6d[...,3:]
but the results I am getting have much larger errors than the ones reported in the paper, which made me think that this is not the correct way to return the regressed pose T_t (which is a bit unintuitive since this pose is combined with the loftr pose). I noticed that there is a condition for self.config['regress']['use_2wt'] that if it is not satisfied a different model is called, that is pose_preds = self.pose_regressor(features). For this model though, if I understand correctly, I would need a different saved model to load. Is this saved model available?

So to refer to my original issue, is it supposed to be wrong to check the quality of the T_t, e.g. by counting inliers, given the default settings of the evaluation on mp3d data? Or, how should I transform the regressed pose from
feats = self.encoder(features)
pred_reg_6d = self.pose_regressor_simple_moe(feats)
so that I get a meaningful pose? I am not referring to the 6d representation here, I am referring to normalization and/or scale, since, as it is, the pose seems to be random, based on the results below, evaluating on the regressed output:

tr rot mean err 31.76 
tr rot median err 24.06 
tr rot pct < 30 57.2 
tr abs mean err 1.7 
tr abs median err 1.03 
tr abs pct < 1 48.4 
rot mean err 14.1 
rot median err 9.33 
rot pct < 30 93.0 
pct successful fits 0.0 
dset size 7996 
auc@5 0.014093276202005406 
auc@10 0.05640453223978019 
auc@20 0.16546028585598807