RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

jinglishi0206
Wed, 25 Oct 2023 02:53:38 +0000

same issue for me when i try to fine-tune llama2 on g5.24xlarge instance using SFTTrainer. but the error did not come back again after i restart notebook and then run training code. did not figure out the reason yet.