[Questions] Some problems in the code

[zafirshi]

Hello, thanks for your wonderful work, I'm having some problems running the project files locally, can u help me out?

1. How to specify the device (CPU or GPU) when run the demo?

   I found that running demo.py directly in the terminal will run with the GPU by default, and the behavior of the CPU run set in the code is not consistent (I found it strange that in Pycharm, Ctrl+Shift+F10 run will use the CPU instead)

   In addition, I found that executing the following line returned different results on the CPU and the GPU:
   

   ZippyPoint/models/matching.py

   Line 19 in e0395fb

   dist_nn, ind_nn = tf.nn.top_k(-1 * dist, 2 if ratio_thresh else 1, sorted=False)

   To solve this problem, I simply set the following conditional judgment statement:

        if "GPU" in dist.backing_device:
        dist_nn = tf.reverse(dist_nn, tf.constant([-1]))
        ind_nn = tf.reverse(ind_nn, tf.constant([-1]))

2. Different inference results on CPU and GPU

   In addition, I found that the same image runs differently on the CPU and GPU (the first one below is the CPU, the second one is the GPU), including the number of keypoints and matches

To reproduce the problem, my environment is configured as follows:

# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_14_21:12:58_PST_2021
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0

# nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03   Driver Version: 510.108.03   CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+

# system
ubuntu 18.04

# some python env
python                    3.8.16
tensorflow                2.10.0
larq                      0.13.0
opencv-python             3.4.14.51
opencv-python             3.4.14.51
opencv-python             3.4.14.51

Thank you again for your outstanding work, can you help me with the above problem? Looking forward to your early reply! :)

[8848hua]

Hi, I'm having some problems, too.
When I was replicating the code, there was no match in the result after running it.
My parameters haven't changed, and the picture hasn't changed.
I tried to change the matching threshold to re-match, but there were many wrong matches. Have you ever encountered such a situation?
I would appreciate it if you would kindly enlighten me.

[zafirshi]

Hello 👋 , I raised this issue about half a year ago, and due to the long time interval, I have forgotten many details.

As far as I remember, in my experiment, the program behaves differently on the GPU and the CPU, which seems to be due to the difference in behavior of the tf.nn.top_k API on the CPU and GPU.

If you haven't gotten the matching pair in my issue, this might be the reason (I guess).

You can try printing and comparing variables in both situations in the _find_nn function. It seems that when running on the GPU, sorted=False is not needed in tf.nn.top_k.

Best wishes.

[8848hua]

Thank you very much for your reply.

I tried to output intermediate variables in the find_nn function and found that dist_nn[..., 0] is smaller than dist_nn[..., 1], so dist_nn[..., 0] should be treated as the nearest neighbor.

But in the code:

          if ratio_thresh:
                    mask = mask & (dist_nn[..., 1] <= (ratio_thresh) * dist_nn[..., 0])

This way of writing seems wrong.
Then, I changed the code to look like this:

if ratio_thresh:
       mask = mask & (dist_nn[..., 0] <= (ratio_thresh) * dist_nn[..., 1])

Also with ratio_thresh set to 0.95, matching points appear, but there are many errors visible to the naked eye.

I'm not sure if there's something I need to change or if there's something wrong with the source code.

The environment I ran was ubuntu20.04, gpu-1080ti.

Now I'll try to dig deeper into the source code and see where the error is.

Thanks again for your reply and best wishes.

[xushangnjlh]

Thank you very much for your reply.

I tried to output intermediate variables in the find_nn function and found that dist_nn[..., 0] is smaller than dist_nn[..., 1], so dist_nn[..., 0] should be treated as the nearest neighbor.

But in the code:

          if ratio_thresh:
                    mask = mask & (dist_nn[..., 1] <= (ratio_thresh) * dist_nn[..., 0])

This way of writing seems wrong. Then, I changed the code to look like this:

if ratio_thresh:
       mask = mask & (dist_nn[..., 0] <= (ratio_thresh) * dist_nn[..., 1])

Also with ratio_thresh set to 0.95, matching points appear, but there are many errors visible to the naked eye.

I'm not sure if there's something I need to change or if there's something wrong with the source code.

The environment I ran was ubuntu20.04, gpu-1080ti.

Now I'll try to dig deeper into the source code and see where the error is.

Thanks again for your reply and best wishes.

I have encounter the same error, have you fixed this?

[8848hua]

Hi, @xushangnjlh，I've taken care of it. The method is to modify the code according to the solution of the user "zafirshi".
Here is my modified code:

dist_nn, ind_nn = tf.nn.top_k(-1 * dist, 2 if ratio_thresh else 1)
if "GPU" in dist.backing_device:
    print('GPU')
    dist_nn = tf.reverse(dist_nn, tf.constant([-1]))
    ind_nn = tf.reverse(ind_nn, tf.constant([-1]))

After I reproduced it, the speed did not reach the efficiency of the paper. If you can successfully reproduce the conclusions in the paper, we can communicate， or if you encounter any problems, we can communicate.

[xushangnjlh]

Hi, @8848hua, I adjust the code accordingly, but the result seems not good, when the viewpoint changes:

My Timer is (GPU):
data=0.005 extraction=0.039 post-process=0.002 matching=0.009

B.T.W if you are a chinese, you can connect me by wechat: 13524833873

[xushangnjlh]

Hi, @8848hua, I adjust the code accordingly, but the result seems not good, when the viewpoint changes:

My Timer is (GPU): data=0.005 extraction=0.039 post-process=0.002 matching=0.009

B.T.W if you are a chinese, you can connect me by wechat: 13524833873

For more comparison with @zafirshi result