ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length

joanplepi
Thu, 03 Nov 2022 09:26:43 +0000

This actually it is the problem, but can we have an explanation why is this the issue? There are two masks, the one coming from running tokenizer on the examples["review_body"], and one coming from running the tokenizer on examples["review_title"]. The first one is not an issue, but the second is and I cannot understand why.