RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc) #38
Description
在运行test时以及train完一个epoch进行val时，都会出现以下错误：
Traceback (most recent call last):
File "tools/test.py", line 273, in
main()
File "tools/test.py", line 237, in main
outputs = single_gpu_test(model, data_loader, args.show, args.show_dir,
File "/home/kumiko/projects/detection/CEASC/mmdet/apis/test.py", line 29, in single_gpu_test
result = model(return_loss=False, rescale=True, **data)
File "/home/kumiko/anaconda3/envs/CEASC/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/kumiko/anaconda3/envs/CEASC/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 50, in forward
return super().forward(*inputs, **kwargs)
File "/home/kumiko/anaconda3/envs/CEASC/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/kumiko/anaconda3/envs/CEASC/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/kumiko/anaconda3/envs/CEASC/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 110, in new_func
return old_func(*args, **kwargs)
File "/home/kumiko/projects/detection/CEASC/mmdet/models/detectors/base.py", line 174, in forward
return self.forward_test(img, img_metas, **kwargs)
File "/home/kumiko/projects/detection/CEASC/mmdet/models/detectors/base.py", line 147, in forward_test
return self.simple_test(imgs[0], img_metas[0], **kwargs)
File "/home/kumiko/projects/detection/CEASC/mmdet/models/detectors/single_stage.py", line 102, in simple_test
results_list = self.bbox_head.simple_test(
File "/home/kumiko/projects/detection/CEASC/mmdet/models/dense_heads/base_dense_head.py", line 341, in simple_test
return self.simple_test_bboxes(feats, img_metas, rescale=rescale)
File "/home/kumiko/projects/detection/CEASC/mmdet/models/dense_heads/dense_test_mixins.py", line 36, in simple_test_bboxes
outs = self.forward(feats)
File "/home/kumiko/projects/detection/CEASC/mmdet/models/dense_heads/anchor_dy_head.py", line 171, in forward
return multi_apply(self.forward_single, feats)
File "/home/kumiko/projects/detection/CEASC/mmdet/core/utils/misc.py", line 30, in multi_apply
return tuple(map(list, zip(*map_results)))
File "/home/kumiko/projects/detection/CEASC/mmdet/models/dense_heads/retina_dy_head.py", line 187, in forward_single
cls_feat, loss = cls_conv((cls_feat, cls_mask, cls_pws))
File "/home/kumiko/anaconda3/envs/CEASC/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/kumiko/anaconda3/envs/CEASC/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forward
input = module(input)
File "/home/kumiko/anaconda3/envs/CEASC/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/kumiko/projects/detection/CEASC/mmdet/models/dense_heads/cuda_dynamic_conv_module.py", line 59, in forward
return self.conv(inputs, mask, (pw[1], pw[2]), gn=self.gn) + pw_back, None
File "/home/kumiko/anaconda3/envs/CEASC/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/kumiko/projects/detection/CEASC/mmdet/models/dense_heads/sparse_conv_net.py", line 166, in forward
fast_ans = self._fast_forward(input, hard, pw, self.weight, self.bias, self.isbias, self.base, gn)
File "/home/kumiko/projects/detection/CEASC/mmdet/models/dense_heads/sparse_conv_net.py", line 153, in _fast_forward
x = Sparse_conv2d.apply(input, hard, weight, bias, self.stride,
File "/home/kumiko/projects/detection/CEASC/mmdet/models/dense_heads/sparse_conv_net.py", line 81, in forward
output = sparse_conv.forward(input, hard, weights, bias, stride[0], padding[0], isbias, base, groups, gnweight, gnbias, pw_mean, pw_rstd, eps, nonzero_hard[0], nonzero_hard[1])[0]
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
除了文件路径外我没有修改任何代码。
请问该如何解决呢？