# Finetuning Scibert and encountering ValueError

**URL:** <https://discuss.huggingface.co/t/finetuning-scibert-and-encountering-valueerror/54034>\
**Category:** Intermediate\
**Created:** [September 8, 2023, 3:40am UTC](https://discuss.huggingface.co/t/finetuning-scibert-and-encountering-valueerror/54034 "2023-09-08T03:40:16Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![kkhan](https://avatars.discourse-cdn.com/v4/letter/k/7bcc69/32.png) [@kkhan](https://discuss.huggingface.co/u/kkhan)\
**Post date:** [September 8, 2023, 3:40am UTC](https://discuss.huggingface.co/t/finetuning-scibert-and-encountering-valueerror/54034/1 "2023-09-08T03:40:16Z")

</div>

I am trying to classify sentences into 5 categories but once I try to train my model i am stuck here because I keep getting this error:

ValueError: The model did not return a loss from the inputs, only the following keys: last\_hidden\_state,pooler\_output. For reference, the inputs it received are input\_ids,token\_type\_ids,attention\_mask.

I have provided he code below

from transformers import AutoModel,Trainer, TrainingArguments, AutoTokenizer  
model = AutoModel.from\_pretrained(“allenai/scibert\_scivocab\_uncased”,num\_labels = 5)  
tokenizer = AutoTokenizer.from\_pretrained(“allenai/scibert\_scivocab\_uncased”)

# Initialize empty lists for tags and text

tags\_list =   
text\_list =

# Iterate through the test\_data list

for entry in train\_data:  
# Extract tags and text from the entry dictionary  
tags = entry[“tags”]  
text = entry[“text”]

```
# Append tags and text to their respective lists
tags_list.append(tags)
text_list.append(text)

```

train\_inputs = tokenizer(text\_list, padding=‘max\_length’, truncation=True,max\_length=72)

class custom\_dataset(torch.utils.data.Dataset):  
def **init** (self, encodings, labels):  
self.encodings = encodings  
self.labels = labels

```
def __getitem__ (self, idx):
    item = {key : torch.tensor(val[idx]) for key, val in self.encodings.items()}
    item['labels'] = torch.tensor(self.labels[idx])
    return item

def __len__ (self):
    return len(self.labels)

```

train\_dataset = custom\_dataset(train\_inputs, tags\_list)

training\_args = TrainingArguments(output\_dir=‘./results’,report\_to = ‘none’)  
trainer = Trainer(  
model=model,  
args=training\_args,  
train\_dataset= train\_dataset,)

trainer.train()

Can someone please help me figure out what i am doing wrong?