I have an unbalanced dataset. When training I want to pass class_weights so the update for rare classes is highen than for large classes. How is this possible in HF with PyTorch? Thanks Philip

Sorry I missed your question and didn’t point you to this directly.

No worries. You did on GitHub… :slight_smile: :+1:

Hi @PhilipMay !Do you mind pasting an example please, as I don’t really understand the documentation

@PhilipMay an example would be highly appreciated.

Overwriting the Trainer can be done as follows (this is also explained in the docs ): from torch import nn from transformers import Trainer class CustomTrainer(Trainer): def compute_loss(self, model, inputs, return_outputs=False): labels = inputs.get("labels") # forward pass …

Thanks a million :slight_smile:

can any help me with how can apply it to text classification? for ex: I have four classes class 1: 200 samples class 2: 100 samples class 3: 20 samples class 4: 10 samples so during predictions, most of my samples are predicting class 1 or 2 because samples are high even I have unlabeled sampl…

Hi @para , the thread above explains exactly what you need to do if you want to use class weights to handle an imbalanced dataset. But if you are not sure what class weights are, what they do or how to overwrite the Trainer yourself I’d suggest that you take more time first to really understand the …

Just to elaborate on the sampling techniques @goerlitz is mentioning: It’s a common ML challenge and the standard approaches for sampling are: Under-sampling: Randomly delete records from the over-represented classes Over-sampling: Duplicate records in the under-represented classes Synthetic samp…

How can I use class_weights when training?

🤗Transformers

PhilipMay September 29, 2020, 10:40am 2

Answering my own question:
Subclass Trainer and override the compute_loss method (see example here).

Handling class imbalance when finetuning a decoder model on text generation

Topic		Replies	Views
Create a weighted loss function to handle imbalance? 🤗Transformers	3	3710	May 21, 2025
Training with class weights 🤗Transformers	5	3341	November 18, 2023
Custom loss weight for train a different weight for validation 🤗Transformers	0	240	April 4, 2024
Cross Entropy Weighted Beginners	12	8286	June 30, 2023
Class weights in Trainer() instance Intermediate	1	785	September 10, 2021

How can I use class_weights when training?

Related topics