RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

bogolese
Mon, 12 Aug 2024 16:49:00 +0000

I’m afraid I’m having the same problem. Mistral 7B Instruct, 2 16G GPUs, BNB 4-bit quantization, runs fine on the first GPU, crashes out with this error on the input dispatched to the second GPU. At this point I am frustrated and running it on only one GPU. FAR less than optimal . . .