Training did not converge

[HTLife]

Recently, I tried to implement VINet[1] and open source it to GitHub HTLife/VINet

I already complete whole network structure, but the network can't converge properly in training.
How could I fix this problem?

Possible problems & solutions:

1. The dataset is too challenging:
   I'm using the EuRoC MAV dataset, which is more challenging than the KITTI VO Dataset used by the DeepVO, Vinet(because the KITTI vehicle image does not shake up and down). NN cannot learn camera movement correctly.

2. Loss function:
   L1 loss is been used and identical to the design in [1]. (I'm not very confident about whether I understand the loss design in [1] currently.) Related code

3. Other hyperparameter problems

---

Chinese translation

VINet 實作，訓練無法收斂，我的網路設計是否有誤？
最近嘗試復現了VINet[1]並將其開源至GitHub HTLife/VINet

目前己經完整整體架構，但在訓練上一直無法正確收斂，想詢問問題出在哪？
推測可能的問題：

1. 資料集太有挑戰性：資料集目前用的是EuRoC MAV dataset，比起DeepVO、VINet等論文使用的KITTI VO Dataset更有挑戰性(因為KITTI車載影像不太會有上下方向晃動)，網路無法正確學習到相機移動

2. Loss function：在HTLife/VINet main.py 中(https://github.com/HTLife/VINet/blob/master/main.py#L210)，目前以L1Loss作為計算方式，加總[1]中提到的兩種loss，這裡我的理解可能不夠充份而實作錯誤。

3. 其它超參數問題

[lrxjason]

How long to train the EuRoc dataset?
Firstly, I try VINet and I found it difficult to converge than I try the DeepVo. It also can't converge. The loss function I use L = pose error + 100 * angle error.

[HTLife]

@lrxjason current version 60d21c7 won't converge.
Which DeepVO GitHub repo did you use?

This implementation of VINet is still under tuning. The current version 60d21c7 still have some part didn't follow the original paper.
Now I'm trying to convert (x y z quaternion ) format to se3(3) and change the loss function, see if this works.

[lrxjason]

@HTLife
I use this https://github.com/sladebot/deepvo/tree/master/src , but I change the training batch part.

I try both the flownetS and CNN. Both of them do not converge.

For the angle, I try pitch roll yaw and qw qx qy qz. But I get the same un-convergence result. And for training 4000 images, I need almost 1 hour for 1 epoch.

[HTLife]

As VINet mentioned, they found it hard to converge by simply training on the frame to frame pose change.
They take accumulated global pose (pose related to the starting point) into account to help NN to converge.
I'll try this idea tomorrow, and verify if this work or not.

@lrxjason Do you have other suggestion to help this network converge?

[lrxjason]

I try just use CNN part( ignore the LSTM part). It also doesn't converge.
I will try to use the PoseNet to train the database tomorrow.

I'm confused about the global pose. If the current camera position is far from the start point which means there is no overlap area how could the global pose work?

From the hardware, I can only set the timestep to 10 which means I can only estimate 10 frames pose. Otherwise, the GPU will out of memory. @HTLife

[HTLife]

About global pose

I think they are not using global pose directly. Instead, they use the "difference" between global pose and accumulated pose. This loss design might reduce the "drift" on estimation.

On page 3999

Code update

I just upload a version with complete SE3 implementation (SHA: 4f14be7).

This is the implementation detail:

Because of the xyzQuaternion feedback design, now the training can only do SGD with batch size 1.

I'm now training with this new implementation and adjusting loss detail.

By the way, @lrxjason did you got any progress on PoseNet?

[HTLife]

"SE3 composition layer" been mentioned in VINet might be related to gvnn. Since the PyTorch implementation is not available (gvnn is implement in torch), I replace the "SE3 composition layer" by SE3 Matrix multiplication.

VINet did not mention the detail of SE3 composition layer, but the related description could be found in [1] (publish by same Lab).

I do not understand the difference between training "directly on se3" and "SE3", and how would that affect the convergence of training.

[1] S.Wang, R.Clark, H.Wen, andN.Trigoni, “End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks,” Int. J. Rob. Res., 2017.

[copark86]

@HTLife I have look at the paper and your code again. It seems like the short batch trajectory structure is important as described in Figure4 of the original paper. SE3 accumulate error as time goes and it will hinder the convergence. It seems like they divided the trajectory into short fragments with different initial poses. Also to keep the continuity they are passing RNN hidden state of the previous batch to the next batch. Maybe it could be the cause of the problem. What do you think?

By the way, how are you debugging the code in the docker? I started python and docker a few days ago. Any advice will be appreciated.

[HTLife]

@copark86 I haven't noticed about passing RNN hidden state to next batch! That might be important.
Also, I'll find some time to replace "matrix multiplication implementation of acc SE3" with "SE3 composition layer". See if this works.

[copark86]

@HTLife I just found that read_R6TrajFile is reading relative pose as 1x6 vector whereas the saved relative pose file is actually written in quaternion format. Is sampleGT_relative.py right code to generate relative pose?

[copark86]

@HTLife Regarding the input of the IMU RNN, I guess it should include every IMU data between two image frames but it seems like your code is loading only 5 nearby IMU data. Please, correct me if I am wrong.

[HTLife]

About the magic number of reading IMU data

@copark86 Sorry for that magic number.
The EuRoC MAV dataset synchronized the camera and imu time. There is one imu record between two images.

(Left: image timestamp, Right: IMU data.csv)

What I did is to feed the RNN with a longer IMU sequence (arbitrary length 5) rather than 3 (image1 time, middle, image2 time).

[HTLife]

About read_R6TrajFile dimension

@copark86
The corresponding part is been showed in the following figure:

1. I calculate the relative pose (x y z ww wx wy wz) from absolute pose (x y z ww wx wy wz)
2. se3 could be represent in 6 value. Therefore, (x y z ww wx wy wz) should convert to R^6, and the output dimension of VINet should also be 6 rather than 7.

I found I read the wrong file. It should be:

self.trajectory_relative = self.read_R6TrajFile('/vicon0/sampled_relative_R6.csv')

[copark86]

@HTLife Regarding IMU data, camera is 20Hz and IMU is 200Hz. The training dataset has 2280 images, 22801 imu measurements which mean there should be 10 imu measurements between two consecutive images. If there are only two imu data between two images as you said, does it mean that you are pre-integrating the IMU data?

Reading sampled_relative_R6 instead of sampled_relative make perfect sense. Thanks for the explanation.

[HTLife]

@copark86 Let me redescribe it.

IMU => no sampling

We should use the raw data of IMU(x,y,z,wx,wy,wz).
The IMU sequence length could be span for 8 images (arbitrary number). We have 70 IMU record between 8 images. So the magic number should change from 5 to 70.

(I assume one red box corresponding to one image)

VICON => sampled to camera rate

The ground truth is VICON, which is 40Hz (twice the frequency of stereo camera 20FPS)