IndexError: Invalid key: 16 is out of bounds for size 0

moschimonsters
Fri, 30 Jun 2023 13:42:53 +0000

For everyone having the problem because of the trl library filtering out data as descibed above, here is the issue regarding this:

github.com/lvwerra/trl

Trainer filters out any data != max length

opened 12:11PM - 26 Jun 23 UTC

KMFODA

Hello, thanks for contributing this amazing repo. I tried using the SFT trainer …a few days ago and found that 95% of my data was being filtered out because it's length was != max_length as per this filter statement: https://github.com/lvwerra/trl/blob/009b82412fe79ea975d2bd72586d03e7fa8724c5/trl/trainer/sft_trainer.py#L296C42-L296C42 Is this the correct filter statement or should it be `if length <= max_length`? If not why do we strip out any data that is != max length? Thanks in advance.