RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

ehalit
Fri, 23 Jun 2023 12:12:59 +0000

Well, this line generates probability estimates for each word which need decoding to generate text. So, for the purpose of demonstration,

print(tokenizer.batch_decode(model(**tokenizer("Hello World", return_tensors="pt")).logits.argmax(-1))[0])

should work. But this means applying greedy decoding and predicting the most probable token without any randomization. Ideally, if you want to generate text for inference (not training) you should use the model.generate() function. Like so,

print(tokenizer.decode(model.generate(**tokenizer("Hello World", return_tensors="pt"))[0]))