# Error running run\_semantic\_segmentation.py

**URL:** <https://discuss.huggingface.co/t/error-running-run-semantic-segmentation-py/18026>\
**Category:** Beginners\
**Created:** [May 17, 2022, 1:03am UTC](https://discuss.huggingface.co/t/error-running-run-semantic-segmentation-py/18026 "2022-05-17T01:03:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nickmuchi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nickmuchi/32/29420_2.png) [@nickmuchi](https://discuss.huggingface.co/u/nickmuchi)\
**Post date:** [May 17, 2022, 1:03am UTC](https://discuss.huggingface.co/t/error-running-run-semantic-segmentation-py/18026/1 "2022-05-17T01:03:02Z")

</div>

Hi there, had issues with running the above script in colab at the following link: [transformers/run\_semantic\_segmentation.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/examples/pytorch/semantic-segmentation/run_semantic_segmentation.py)

Training args I used are below:

!python /content/drive/MyDrive/run\_semantic\_segmentation.py   
–model\_name\_or\_path nvidia/mit-b5   
–dataset\_name nickmuchi/rugd-dataset-all   
–output\_dir /content/drive/MyDrive/segformer-finetuned-rugd-out   
–remove\_unused\_columns False   
–do\_train   
–do\_eval   
–evaluation\_strategy steps   
–push\_to\_hub   
–push\_to\_hub\_model\_id segformer-finetuned-rugd   
–max\_steps 10000   
–learning\_rate 0.00006   
–lr\_scheduler\_type polynomial   
–per\_device\_train\_batch\_size 2   
–per\_device\_eval\_batch\_size 2   
–logging\_strategy steps   
–logging\_steps 100   
–evaluation\_strategy epoch   
–save\_strategy epoch   
–save\_total\_limit 2   
–load\_best\_model\_at\_end True   
–seed 1337   
–max\_train\_samples 3000   
–max\_eval\_samples 500

Error:

Traceback (most recent call last):  
File “/content/drive/MyDrive/run\_semantic\_segmentation.py”, line 508, in   
main()  
File “/content/drive/MyDrive/run\_semantic\_segmentation.py”, line 483, in main  
train\_result = trainer.train(resume\_from\_checkpoint=checkpoint)  
File “/usr/local/lib/python3.7/dist-packages/transformers/trainer.py”, line 1324, in train  
ignore\_keys\_for\_eval=ignore\_keys\_for\_eval,  
File “/usr/local/lib/python3.7/dist-packages/transformers/trainer.py”, line 1559, in \_inner\_training\_loop  
tr\_loss\_step = self.training\_step(model, inputs)  
File “/usr/local/lib/python3.7/dist-packages/transformers/trainer.py”, line 2206, in training\_step  
loss.backward()  
File “/usr/local/lib/python3.7/dist-packages/torch/\_tensor.py”, line 363, in backward  
torch.autograd.backward(self, gradient, retain\_graph, create\_graph, inputs=inputs)  
File “/usr/local/lib/python3.7/dist-packages/torch/autograd/init.py”, line 175, in backward  
allow\_unreachable=True, accumulate\_grad=True) # Calls into the C++ engine to run the backward pass  
RuntimeError: CUDA error: CUBLAS\_STATUS\_EXECUTION\_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)  
0% 0/10000 [00:00\<?, ?it/s]

Tried googling but did not get much, thanks.

---

<div class="post-metadata">

**Author:** ![nickmuchi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nickmuchi/32/29420_2.png) [@nickmuchi](https://discuss.huggingface.co/u/nickmuchi)\
**Post date:** [May 20, 2022, 11:21am UTC](https://discuss.huggingface.co/t/error-running-run-semantic-segmentation-py/18026/2 "2022-05-20T11:21:14Z")

</div>

Hi @nielsr could you please assist as you recently did the demo? Thanks

---

<div class="post-metadata">

**Author:** ![cog](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@cog](https://discuss.huggingface.co/u/cog)\
**Post date:** [May 23, 2022, 8:03am UTC](https://discuss.huggingface.co/t/error-running-run-semantic-segmentation-py/18026/3 "2022-05-23T08:03:05Z")

</div>

hi nickmuchi.

if you see more detail of error, use this code at front of your codes.

`os.environ['CUDA_LAUNCH_BLOCKING'] = "1"`

there is so many error case that rising CUDA error.

here is some case i faced before. i hope its help you.

**1. input dimension mismatch.**  
layer shape is (512,1) but i put (512,13), 13 is my custom data label\_num.

**2. input data error**  
you need check input data’s tensor range is correct.  
my data was [-1,1,1,1] but range is (0~75), it rising cuda, cublas error.

**3. version conflict**  
I do not recommend this case, but I write it just in case.  
if you install your env with official document’s requirements, pass it.  
but if not, check lib’s version.  
there is almost conflict occur by cuda, cuddn, GPU.  
also pytorch, transformer, etc lib version need to check.

regards