# BLEU evaluation with multiple references

**URL:** <https://discuss.huggingface.co/t/bleu-evaluation-with-multiple-references/19048>\
**Category:** 🤗Datasets\
**Created:** [June 12, 2022, 12:17pm UTC](https://discuss.huggingface.co/t/bleu-evaluation-with-multiple-references/19048 "2022-06-12T12:17:16Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sunhaozhepy](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@sunhaozhepy](https://discuss.huggingface.co/u/sunhaozhepy)\
**Post date:** [June 12, 2022, 12:17pm UTC](https://discuss.huggingface.co/t/bleu-evaluation-with-multiple-references/19048/1 "2022-06-12T12:17:16Z")

</div>

Hi,

I’m trying to train a T5 model on a seq2seq task. The dataset has multiple ground truths for the generation; I split the references to get more training data, and I want to validate and test with all references to calculate the BLEU score, and for validation I want to save the model with the highest BLEU score calculated on the validation set. Now this has two problems:

1. the common `DataCollatorForSeq2Seq` can’t deal with that because the label is 3-dimensional: the following is an example:)

```auto
'references': [[tensor([3613, 8, 4963, 13, 8, 4033, 1]),
   tensor([320, 21, 3, 9, 9717, 2195, 17041, 1]),
   tensor([661, 550, 45, 8, 3, 25895, 3797, 1])],
  [tensor([ 34, 808, 3, 13287, 5600, 11, 3, 29, 6833, 81,
             460, 676, 12, 129, 12, 1455, 5, 1]),
   tensor([ 34, 47, 3412, 53, 7501, 116, 3, 29, 6833, 3030, 12, 129,
             95, 5, 1]),
   tensor([ 8, 1282, 1969, 263, 34, 1256, 21, 9635, 52, 7, 12, 253,
              3, 29, 6833, 5, 1])],
  [tensor([ 79, 261, 16352, 7, 12, 199, 2331, 9321, 7, 5,
               1]),
   tensor([79, 2139, 7208, 7479, 70, 9321, 7, 5, 1]),
   tensor([79, 356, 95, 46, 1470, 718, 1131, 2269, 5, 1])]]}

```

1. I don’t know how to use this configuration in the `Trainer` API: is there a way not to calculate the validation loss, and only calculate the BLEU score?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [July 4, 2022, 2:09pm UTC](https://discuss.huggingface.co/t/bleu-evaluation-with-multiple-references/19048/2 "2022-07-04T14:09:06Z")

</div>

Hi ! Feel free to questions about the `Trainer` API in the `🤗Transformers` section 😉

Have you tried taking a look at the `compute_metrics` argument of the Seq2SeqTrainer ?

> **[Trainer](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Seq2SeqTrainer)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![sunhaozhepy](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@sunhaozhepy](https://discuss.huggingface.co/u/sunhaozhepy)\
**Post date:** [July 5, 2022, 7:55am UTC](https://discuss.huggingface.co/t/bleu-evaluation-with-multiple-references/19048/3 "2022-07-05T07:55:06Z")

</div>

Hi,

Thanks for your reply! Basically I’ve decided not to evaluate during training, but to evaluate my model after training on each checkpoint.