# Getting torch.cuda.halfTensor error while using DeepSpeed with accelerate

**URL:** <https://discuss.huggingface.co/t/getting-torch-cuda-halftensor-error-while-using-deepspeed-with-accelerate/39997>\
**Category:** 🤗Accelerate\
**Created:** [May 15, 2023, 9:05pm UTC](https://discuss.huggingface.co/t/getting-torch-cuda-halftensor-error-while-using-deepspeed-with-accelerate/39997 "2023-05-15T21:05:53Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![sgauravm](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sgauravm](https://discuss.huggingface.co/u/sgauravm)\
**Post date:** [May 15, 2023, 9:05pm UTC](https://discuss.huggingface.co/t/getting-torch-cuda-halftensor-error-while-using-deepspeed-with-accelerate/39997/1 "2023-05-15T21:05:54Z")

</div>

I am trying to train a GPT-2 large language model using Deepspeed plugin of accelerate. Every time when the script calls this line,  
`loss = model(torch.stack(batch['input_ids']).T,labels=torch.stack(batch['input_ids']).T).loss` , I get the following error:  
`RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.HalfTensor instead (while checking arguments for embedding) inputs_embeds = self.wte(input_ids)`.  
I also checked the data type of `torch.stack(batch['input_ids']).T`. It is torch.Long.

For reference I am providing my accelerate config file parameters and a more descriptive error stacktrace.  
Configfile:

```auto
deepspeed_config:

gradient_accumulation_steps: 1

gradient_clipping: 1.0

offload_optimizer_device: none

offload_param_device: none

zero3_init_flag: true

zero_stage: 2

distributed_type: DEEPSPEED

fsdp_config: {}

machine_rank: 0

main_process_ip: null

main_process_port: null

main_training_function: main

mixed_precision: fp16

num_machines: 1

num_processes: 4

use_cpu: false

```

Detailed error:

```auto
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-48d0be80-7221-441b-911a-c6bb2ef50dc5/lib/python3.9/site-packages/torch/nn/functional.py", line 2210, in embedding
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-48d0be80-7221-441b-911a-c6bb2ef50dc5/lib/python3.9/site-packages/torch/nn/functional.py", line 2210, in embedding
transformer_outputs = self.transformer(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-48d0be80-7221-441b-911a-c6bb2ef50dc5/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-48d0be80-7221-441b-911a-c6bb2ef50dc5/lib/python3.9/site-packages/torch/nn/modules/sparse.py", line 160, in forward
return F.embedding(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-48d0be80-7221-441b-911a-c6bb2ef50dc5/lib/python3.9/site-packages/torch/nn/functional.py", line 2210, in embedding
return forward_call(*input, **kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-48d0be80-7221-441b-911a-c6bb2ef50dc5/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 843, in forward
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeErrorreturn torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse):
Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.HalfTensor instead (while checking arguments for embedding)
RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.HalfTensor instead (while checking arguments for embedding)
inputs_embeds = self.wte(input_ids)

```

---

<div class="post-metadata">

**Author:** ![hsuyab](https://avatars.discourse-cdn.com/v4/letter/h/da6949/32.png) [@hsuyab](https://discuss.huggingface.co/u/hsuyab)\
**Post date:** [May 16, 2023, 6:39pm UTC](https://discuss.huggingface.co/t/getting-torch-cuda-halftensor-error-while-using-deepspeed-with-accelerate/39997/3 "2023-05-16T18:39:10Z")

</div>

- Can you share your code or some re-usable example to test this.
- Also can you see what is being passed into your input when you are running by printing that.

---

<div class="post-metadata">

**Author:** ![sgauravm](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sgauravm](https://discuss.huggingface.co/u/sgauravm)\
**Post date:** [May 16, 2023, 8:58pm UTC](https://discuss.huggingface.co/t/getting-torch-cuda-halftensor-error-while-using-deepspeed-with-accelerate/39997/4 "2023-05-16T20:58:11Z")

</div>

Hi @hsuyab thanks for replying. yes I tested the output by printing. It is torch.Long(). I think since it is ‘fp16’ deepseed is converting the input\_ids to torch.cuda.half\_tensor. I cannot share the code as it is my comapny’s internal code but what I can share is the output of ds\_report. Is it possible for you to check whether this is okay:

```auto
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING] async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING] async_io: please install the libaio-dev package with apt
 [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING] sparse_attn requires a torch version >= 1.5 but detected 2.0
 [WARNING] using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-484d6d7c-f559-4582-aa77-2c164373abce/lib/python3.9/site-packages/torch']
torch version .................... 2.0.1+cu117
deepspeed install path ........... ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-484d6d7c-f559-4582-aa77-2c164373abce/lib/python3.9/site-packages/deepspeed']
deepspeed info ................... 0.8.3, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.3
deepspeed wheel compiled w. ...... torch 1.13, cuda 11.7

```

---

<div class="post-metadata">

**Author:** ![madscientist1224](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@madscientist1224](https://discuss.huggingface.co/u/madscientist1224)\
**Post date:** [May 18, 2023, 7:18pm UTC](https://discuss.huggingface.co/t/getting-torch-cuda-halftensor-error-while-using-deepspeed-with-accelerate/39997/5 "2023-05-18T19:18:00Z")

</div>

Having this exact problem. Were you able to solve?

---

<div class="post-metadata">

**Author:** ![madscientist1224](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@madscientist1224](https://discuss.huggingface.co/u/madscientist1224)\
**Post date:** [May 18, 2023, 7:38pm UTC](https://discuss.huggingface.co/t/getting-torch-cuda-halftensor-error-while-using-deepspeed-with-accelerate/39997/6 "2023-05-18T19:38:58Z")

</div>

I solved by changing accelerate config to not use fp16 (stage 2)

---

<div class="post-metadata">

**Author:** ![sgauravm](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sgauravm](https://discuss.huggingface.co/u/sgauravm)\
**Post date:** [May 19, 2023, 1:03am UTC](https://discuss.huggingface.co/t/getting-torch-cuda-halftensor-error-while-using-deepspeed-with-accelerate/39997/7 "2023-05-19T01:03:33Z")

</div>

Yeah, it works by removing fp16, but then Out of Memory error start coming. FP16 would be required for larger batch sizes. I am also stuck and unable to find a solution. I will probably stop using accelerate and directly use DeepSpeed library with pytorch.

---

<div class="post-metadata">

**Author:** ![yuanye](https://avatars.discourse-cdn.com/v4/letter/y/977dab/32.png) [@yuanye](https://discuss.huggingface.co/u/yuanye)\
**Post date:** [June 8, 2023, 1:53am UTC](https://discuss.huggingface.co/t/getting-torch-cuda-halftensor-error-while-using-deepspeed-with-accelerate/39997/8 "2023-06-08T01:53:48Z")

</div>

hi，i fixed this problem by change my deepspeed config file.  
They key is set \<auto\_cast\> to false, then you will find there no longer has the problem of “LongTensor be tranfered to HalfTensor”

---

<div class="post-metadata">

**Author:** ![yuanye](https://avatars.discourse-cdn.com/v4/letter/y/977dab/32.png) [@yuanye](https://discuss.huggingface.co/u/yuanye)\
**Post date:** [June 8, 2023, 1:54am UTC](https://discuss.huggingface.co/t/getting-torch-cuda-halftensor-error-while-using-deepspeed-with-accelerate/39997/9 "2023-06-08T01:54:50Z")

</div>

you can solve it in my answer

---

<div class="post-metadata">

**Author:** ![smangrul](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smangrul/32/3791_2.png) [@smangrul](https://discuss.huggingface.co/u/smangrul)\
**Post date:** [June 23, 2023, 9:47am UTC](https://discuss.huggingface.co/t/getting-torch-cuda-halftensor-error-while-using-deepspeed-with-accelerate/39997/10 "2023-06-23T09:47:40Z")

</div>

is this still an issue?