Error on running test code

[AliBuildsAI]

When I run test_net.py, I encounter CUDA memory related errors (e.g. segmentation fault,
CUDA error: an illegal memory access was encountered, etc). Error messages change from time to time.
Anyone with the similar problems?

[kevinkit]

What kind of GPU do you use? DA-RNN needs at least 6gb I think. However, it may be related to other issues which different third party libraries that need to be installed correctly, see #2 / #10 .
Also what kind of CUDA, cuDNN, TensorFlow and Ubuntu are you using?

[AliBuildsAI]

I ran the code for training with no problem, so there is probably no problem with dependencies.
I have a TITAN X and a Geforce GTX gpu.
CUDA version: 8.0.61
CuDNN: 5.1
Ubuntu: 16.04
Tensorflow version: 1.2.1

[kevinkit]

Do you give the device ID as an input parameter to your script?

Check with nvidia-smi the ID of your Titan GPU and parse it to the script. I do not know which kind of GeForce GTX GPU you have, but a TITAN should run just fine. (However, the test script did not work yet, cause of #9 )

Btw, the training scripts have not been an issue ever, while the test scripts seems to be the trouble maker.

[AliBuildsAI]

Yes, the device ID is 0.
This is the command I ran:
./experiments/scripts/rgbd_scene_multi_rgbd_test.sh

And here is the inside of rgbd_scene_multi_rgbd_test.sh:

#!/bin/bash

set -x
set -e

export PYTHONUNBUFFERED="True"
export CUDA_VISIBLE_DEVICES=$1
#export LD_PRELOAD=/usr/lib/libtcmalloc.so.4

LOG="experiments/logs/rgbd_scene_multi_rgbd.txt.date +'%Y-%m-%d_%H-%M-%S'"
exec &> >(tee -a "$LOG")
echo Logging output to "$LOG"

train FCN for multiple frames

time ./tools/train_net.py --gpu 0
--network vgg16
--weights data/imagenet_models/vgg16_convs.npy
--imdb rgbd_scene_train
--cfg experiments/cfgs/rgbd_scene_multi_rgbd.yml
--iters 40000

if [ -f $PWD/output/rgbd_scene/rgbd_scene_val/vgg16_fcn_rgbd_multi_frame_rgbd_scene_iter_40000/segmentations.pkl ]
then
rm $PWD/output/rgbd_scene/rgbd_scene_val/vgg16_fcn_rgbd_multi_frame_rgbd_scene_iter_40000/segmentations.pkl
fi

test FCN for multiple frames

time ./tools/test_net.py --gpu 0
--network vgg16
--model output/rgbd_scene/rgbd_scene_train/vgg16_fcn_rgbd_multi_frame_rgbd_scene_iter_40000.ckpt
--imdb rgbd_scene_val
--cfg experiments/cfgs/rgbd_scene_multi_rgbd.yml
--rig data/RGBDScene/camera.json
--kfusion 1

[kevinkit]

have you tried running: ./experiments/scripts/rgbd_scene_multi_rgbd_test.sh 0 instead?

[kevinkit]

maybe try running it with sudo

[yuxng]

The testing code calls the c++ KinectFusion library in Python. This step is not stable. I also encountered crashes, due to some malloc issue inside python. You can debug by running "gdb --args python ./tools/test_net.py --gpu 0 --network vgg16 --model output/rgbd_scene/rgbd_scene_train/vgg16_fcn_rgbd_multi_frame_rgbd_scene_iter_40000.ckpt
--imdb rgbd_scene_val --cfg experiments/cfgs/rgbd_scene_multi_rgbd.yml --rig data/RGBDScene/camera.json", and backtrace to see the problem.

[AliBuildsAI]

I ran this and there was no problem, but when I added --kfusion 1 at the end, I encountered this error:

[New Thread 0x7ffe65ffb700 (LWP 8553)]
[New Thread 0x7ffe667fc700 (LWP 8554)]
[New Thread 0x7ffe67fff700 (LWP 8555)]
[New Thread 0x7ffe677fe700 (LWP 8556)]
[New Thread 0x7ffe66ffd700 (LWP 8557)]
[New Thread 0x7ffe5e22a700 (LWP 8558)]
[New Thread 0x7ffe5da29700 (LWP 8559)]
[New Thread 0x7ffe5d228700 (LWP 8560)]
[New Thread 0x7ffe5ca27700 (LWP 8561)]
[New Thread 0x7ffe4ffff700 (LWP 8562)]
[New Thread 0x7ffe4f7fe700 (LWP 8563)]
[New Thread 0x7ffe4effd700 (LWP 8564)]
[New Thread 0x7ffe4e7fc700 (LWP 8565)]
[New Thread 0x7ffe4dffb700 (LWP 8566)]
[New Thread 0x7ffe4d7fa700 (LWP 8567)]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
__memmove_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:245
245 ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S: No such file or directory.
(gdb) quit
A debugging session is active.

Inferior 1 [process 7937] will be killed.

[AliBuildsAI]

@kevinkit the same happens when I add 0 at the end of the command.

When I ran it with sudo, this error happens:

• set -e
• export PYTHONUNBUFFERED=True
• PYTHONUNBUFFERED=True
• export CUDA_VISIBLE_DEVICES=0
• CUDA_VISIBLE_DEVICES=0
  ++ date +%Y-%m-%d_%H-%M-%S
• LOG=experiments/logs/rgbd_scene_multi_rgbd_test.txt.2017-08-15_17-08-27
• exec
  ++ tee -a experiments/logs/rgbd_scene_multi_rgbd_test.txt.2017-08-15_17-08-27
• echo Logging output to experiments/logs/rgbd_scene_multi_rgbd_test.txt.2017-08-15_17-08-27
  Logging output to experiments/logs/rgbd_scene_multi_rgbd_test.txt.2017-08-15_17-08-27
• '[' -f /home/aliman/DA-RNN-master/output/rgbd_scene/rgbd_scene_val/vgg16_fcn_rgbd_multi_frame_rgbd_scene_iter_40000/segmentations.pkl ']'
• ./tools/test_net.py --gpu 0 --network vgg16 --model data/fcn_models/rgbd_scene/vgg16_fcn_rgbd_multi_frame_rgbd_scene_iter_40000.ckpt --imdb rgbd_scene_val --cfg experiments/cfgs/rgbd_scene_multi_rgbd.yml --rig data/RGBDScene/camera.json --kfusion 1
  Traceback (most recent call last):
  File "./tools/test_net.py", line 13, in
  from fcn.test import test_net
  File "/home/aliman/DA-RNN-master/tools/../lib/fcn/test.py", line 25, in
  from kinect_fusion import kfusion
  ImportError: libkfusion.so: cannot open shared object file: No such file or directory

(But I have libkfusion.so in DA-RNN/lib/kinect_fusion/build directory)

[doomxhc]

Have you ever solved the problem?I encounter the same situation and I don't know how to work it our

[kevinkit]

Like mentioned by @yuxng before, you can try to backtrace the problem with the gdb debugger, with the command like mentioned before:

"You can debug by running "gdb --args python ./tools/test_net.py --gpu 0 --network vgg16 --model output/rgbd_scene/rgbd_scene_train/vgg16_fcn_rgbd_multi_frame_rgbd_scene_iter_40000.ckpt
--imdb rgbd_scene_val --cfg experiments/cfgs/rgbd_scene_multi_rgbd.yml --rig data/RGBDScene/camera.json", and backtrace to see the problem."

[lizhihuit]

@AliManUtd1993 ,do you compile the DA-RNN succesful? I always encounter the error in Kinect_Fusion

[AliBuildsAI]

I compiled all parts except kinect_fusion part.

[baolinv0]

@AliManUtd1993 , do you compile the DA-RNN succesfully now? When I test_kinect_fusion.sh , it always show

ImportError: libkfusion.so: cannot open shared object file: No such file or directory

But libkfusion.so is in lib/kinect_fusion/build. And others can run succesfully.

[AliBuildsAI]

No, I did not try anymore.