IndexError: Invalid key: 16 is out of bounds for size 0

maipp
Sat, 08 Jul 2023 23:07:02 +0000

The problem here is with the “_prepare_dataset()” function in “trl/trainer/sft_trainer.py”. Instead, what the source code recommended was using “trl.trainer.ConstantLengthDataset” to create datasets.

I changing my dataset to ConstantLengthDataset, removing “group_by_length”, “dataset_text_field” and added “packing=True”.

from trl import SFTTrainer
from trl.trainer import ConstantLengthDataset

dataset = ConstantLengthDataset(
    tokenizer=tokenizer,
    dataset=dataset,
    dataset_text_field="text")

training_arguments = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=per_device_train_batch_size,
    gradient_accumulation_steps=gradient_accumulation_steps,
    optim=optim,
    save_steps=save_steps,
    logging_steps=logging_steps,
    learning_rate=learning_rate,
    fp16=True,
    max_grad_norm=max_grad_norm,
    max_steps=max_steps,
    warmup_ratio=warmup_ratio,
  # group_by_length=True,
    lr_scheduler_type=lr_scheduler_type,
)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=peft_config,
  # dataset_text_field="text",
    max_seq_length=max_seq_length,
    tokenizer=tokenizer,
    args=training_arguments,
    packing=True # added this
)

Now the code should work.