torch.FatalError: function_attributes(): after cudaFuncGetAttributes: invalid device function #26
Description
The following error occurred when I ran the train_mobile_mvd:
/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/cuda/init.py:95: UserWarning:
Found GPU0 Tesla V100-SXM2-16GB which requires CUDA_VERSION >= 9000 for
optimal performance and fast startup time, but your PyTorch was compiled
with CUDA_VERSION 8000. Please install the correct PyTorch binary
using instructions from http://pytorch.org
warnings.warn(incorrect_binary_warn % (d, name, 9000, CUDA_VERSION))
^M 0%| | 0/9000 [00:00<?, ?it/s]['/workfs/lhhuang/segment/LightNet-master2/LightNet-master/scripts', '/workfs/lhhuang/segment/LightNet-master2/LightNet-master/scripts', '/dat01/lhhuang/models/research', '/dat01/lhhuang/models/research/slim', '/dat01/lhhuang/models/research/delf', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python36.zip', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/lib-dynload', '/home/lhhuang/.local/lib/python3.6/site-packages', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages', '/workfs/lhhuang/segment/LightNet-master2/LightNet-master/datasets', '/workfs/lhhuang/segment/LightNet-master2/LightNet-master']
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
- Setting up DataLoader...
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
There are 66 labels in the config file
Found 18000 training images...
There are 66 labels in the config file
Found 2000 validation images...+++++++++++++++++++++++++++++++++++++++++++++++++++++++
- Setting up Model...
- Model Training start...
^M 0%| | 1/9000 [00:04<12:23:13, 4.96s/it]^M> Epoch [1/256]: 0%| | 1/9000 [00:04<12:23:13, 4.96s/it]Traceback (most recent call last):
File "train_mobile_mvd.py", line 348, in
train(train_args, data_path, save_path)
File "train_mobile_mvd.py", line 185, in train
out_stg1, out_stg2, out_stg3 = model(images) # Here we have 3 output for 3 loss
File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/models/mobilenetv2plus.py", line 169, in forward
de_stg1 = self.aspp(stg8)[1]
File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 67, in forward
input = module(input)
File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/misc.py", line 334, in forward
x = self.in_norm(x)
File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/bn.py", line 96, in forward
self.activation, self.slope)
File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/functions.py", line 97, in forward
_act_forward(ctx, x)
File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/functions.py", line 41, in _act_forward
_ext.leaky_relu_cuda(x, ctx.slope)
File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/utils/ffi/init.py", line 180, in safe_call
result = torch._C._safe_call(*args, **kwargs)
torch.FatalError: function_attributes(): after cudaFuncGetAttributes: invalid device function