IndexError: Invalid key: 16 is out of bounds for size 0

deepak11
Tue, 14 Mar 2023 07:46:20 +0000

I am also facing the same issue, check the below code.

from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments
import os
from datasets import load_dataset
os.environ[“WANDB_DISABLED”] = “true”

tokenizer = AutoTokenizer.from_pretrained(“EleutherAI/gpt-j-6B”, revision=“float16”, low_cpu_mem_usage=True)
model = AutoModelForCausalLM.from_pretrained(“EleutherAI/gpt-j-6B”, revision=“float16”, low_cpu_mem_usage=True)
train_dataset = load_dataset(‘D:\Vinoth\Finetune_GPTNEO_GPTJ6B\cp’)

training_args = TrainingArguments(
output_dir=‘results’,
num_train_epochs=1,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
warmup_steps=500,
weight_decay=0.01,
logging_dir=‘logs’,

)

trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset[“train”]

)

trainer.train()