ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length

popaqy
Tue, 01 Nov 2022 15:25:39 +0000

You have this preprocess function, right?

def preprocess_function(examples):
    model_inputs = tokenizer(
        examples["review_body"],
        max_length=max_input_length,
        truncation=True,
    )
    labels = tokenizer(examples["review_title"], max_length=max_target_length, truncation=True)
    model_inputs["labels"] = labels["input_ids"]
    model_inputs["labels_mask"] = labels["attention_mask"]
    return model_inputs

All you have to do is to remove the line:
model_inputs["labels_mask"] = labels["attention_mask"]

So at the end you end up with:

def preprocess_function(examples):
    model_inputs = tokenizer(
        text = examples["review_body"],
        max_length=max_input_length,
        truncation=True,
    )
    labels = tokenizer(examples["review_title"], max_length=max_target_length, truncation=True)
    model_inputs["labels"] = labels["input_ids"]
    return model_inputs