IndexError: Invalid key: 16 is out of bounds for size 0

moschimonsters
Thu, 29 Jun 2023 08:05:19 +0000

@envizzion I get the same error when replacing the timdettmers/openassistant-guanaco dataset used in the finetuning code. I created a custom dataset oriented at the format of the openassistant, saved it as jsonl, then loaded it with datasets and saved in arrow format.
When i load from disk:

# downloaded openassistant dataset
dataset1 = load_from_disk('./data/out/openassistant')
# mine
dataset2 = load_from_disk('./data/out/combined')

print(dataset1)
print(dataset2)

print(dataset1[0])
print(dataset2[0])

it looks like this:

Dataset({
    features: ['text'],
    num_rows: 9846
})
Dataset({
    features: ['text'],
    num_rows: 1200
})
{'text': '### Human: Can you write a short introduction about the relevance of the term "monopsony" in economics? Please use examples related to potential monopsonies in the labour market and cite relevant research.### Assistant: "Monopsony" refers to a market structure where there is only one buyer for a particular good or service. In economics, this term is particularly relevant in the labor market, where a monopsony employer has significant power over the wages and working conditions of their employees. The presence of a monopsony can result in lower wages and reduced employment opportunities for workers, as the employer has little incentive to increase wages or provide better working conditions.\n\nRecent research has identified potential monopsonies in industries such as retail and fast food, where a few large companies control a significant portion of the market (Bivens & Mishel, 2013). In these industries, workers often face low wages, limited benefits, and reduced bargaining power, leading to a situation where they are dependent on the employer for their livelihood. This dependence can result in further suppression of wages and a decline in working conditions.\n\nOverall, the concept of monopsony is essential to understanding the dynamics of labor markets and the impact of market power on workers. Further research is needed to understand the extent and impact of monopsonies on the economy and to develop policies to address this issue.\n\nReferences:\nBivens, J., & Mishel, L. (2013). The Pay of Corporate Executives and Financial Professionals as Evidence of Rents in Top 1 Percent Incomes. Journal of Economic Perspectives, 27(3), 57-78.### Human: Now explain it to a dog'}
{'text': 'Frage: Warum sind geschätzte und tatsächliche Kosten bei Bauprojekten ein zentrales Thema? Antwort: Die geschätzten und tatsächlichen Kosten sind bei vielen Bauprojekten ein zentrales Thema, da der Bauherr eine hohe Investitionssumme einsetzen muss, welche in vielen Fällen die sonstigen Ausgaben um ein Vielfaches übersteigt.'}

Using the openassistant dataset the training runs fine but using my own i get the same “IndexError: Invalid key: 0 is out of bounds for size 0.”.

Edit1: From the code in datasets/formatting/formatting.py i guess that the problem seems to be the size calculation in line 582. I think it shouldn’t be 0. The index check raises the error correctly since key>= size. see: datasets/src/datasets/formatting/formatting.py at 0d2b8854c265b4dc202e480427890f472b34ea15 · huggingface/datasets · GitHub

Edit2: In my case the problem is definitely the dataset structure of my own dataset. The column “text” gets removed by the _remove_unused_columns function of the transformers library trainer (trainer.py). Just for comparison i printed out the column names in the function for the openassistant dataset the data seems to be in “input_ids” while in mine it landed in “text”

dataset column names: ['input_ids']
signature_columns: ['input_ids', 'attention_mask', 'inputs_embeds', 'labels', 'output_attentions', 'output_hidden_states', 'return_dict', 'kwargs', 'label', 'label_ids', 'labels']
ignored_columns: []
columns: ['input_ids']

dataset column names: ['text']
signature_columns: ['input_ids', 'attention_mask', 'inputs_embeds', 'labels', 'output_attentions', 'output_hidden_states', 'return_dict', 'kwargs', 'label_ids', 'label', 'labels']
ignored_columns: ['text']
columns: []