THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=11 : invalid argument

[theshy456]

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=11 : invalid argument
Traceback (most recent call last):
File "./tools/trainval_net.py", line 135, in
max_iters=args.max_iters)
File "/hy-tmp/import_datasets/CASD-master_2022-01-20-210309/CASD-master/tools/../lib/model/train_val.py", line 377, in train_net
sw.train_model(max_iters)
File "/hy-tmp/import_datasets/CASD-master_2022-01-20-210309/CASD-master/tools/../lib/model/train_val.py", line 291, in train_model
cls_det_loss, refine_loss_1, refine_loss_2, consistency_loss, total_loss = self.net.train_step(blobs,self.optimizer,iter)
File "/hy-tmp/import_datasets/CASD-master_2022-01-20-210309/CASD-master/tools/../lib/nets/network.py", line 634, in train_step
self.forward(blobs['data'], blobs['image_level_labels'], blobs['im_info'], blobs['gt_boxes'], blobs['ss_boxes'], step)
File "/hy-tmp/import_datasets/CASD-master_2022-01-20-210309/CASD-master/tools/../lib/nets/network.py", line 562, in forward
roi_labels_1, keep_inds_1, roi_labels_2, keep_inds_2, bbox_pred, rois = self._predict_train(ss_boxes_all, step)
File "/hy-tmp/import_datasets/CASD-master_2022-01-20-210309/CASD-master/tools/../lib/nets/network.py", line 508, in _predict_train
roi_labels_2, keep_inds_2, bbox_pred = self._region_classification_train(pool5_roi, fc7_roi,fc7_context, fc7_frame, step)
File "/hy-tmp/import_datasets/CASD-master_2022-01-20-210309/CASD-master/tools/../lib/nets/network.py", line 398, in _region_classification_train
mask_1 = self._inverted_attention(bbox_feats_new, gt, keep_inds_1_new, 1, step, fg_num_1_new, bg_num_1_new)
File "/hy-tmp/import_datasets/CASD-master_2022-01-20-210309/CASD-master/tools/../lib/nets/network.py", line 147, in _inverted_attention
pooled_feat_before_after = torch.cat((bbox_feats_new, bbox_feats_new * mask_all), dim=0)
RuntimeError: CUDA out of memory. Tried to allocate 766.00 MiB (GPU 0; 10.76 GiB total capacity; 9.17 GiB already allocated; 658.56 MiB free; 118.66 MiB cached)