IndexError: Invalid key: 16 is out of bounds for size 0

arnoldmatt
Wed, 05 Jun 2024 10:47:06 +0000

Hi sfalk,

The error “IndexError: Invalid key: 16 is out of bounds for size 0” suggests your NewDataset might be empty. The training loop tries to access the 16th element (index 15), but there are zero elements in the dataset.

Double-check your _generate_examples function to ensure it’s yielding data correctly. You can also try printing the number of examples generated during the initial creation to verify.

Here are some additional tips for avoiding caching issues:

Consider using yield from to stream data directly from your generation process instead of building the entire dataset in memory.
If you must cache, explore libraries like dask or ray for parallel processing and memory management.