I use data_set = load_dataset(self.data_file_path, cache_dir=cache_dir, split=“train”) Report when loading dataset (approximately 84GB) pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 10761561509， Try setting up according to the help provided in other po…

Apparently, this is an issue with PyArrow, and although some of it has been resolved, it still seems to be unresolved. @lhoestq Loading big dataset raises pyarrow.lib.ArrowNotImplementedError opened 02:42PM - 02 Apr 23 UTC (UTC) …

Yes, I have seen similar posts with the same issue: Minhash Deduplication - #11 by conceptofmind But I tried this method and it didn’t solve the error. May I ask if there is any way you can help me solve this problem? Thank you!

How about trying .shard()? [image] How to load a large hf dataset efficiently? 🤗Datasets Thank you for the reply, following your suggestion is there any way in hf dataset, split into multiple shards to be loaded in parallel?

Thank you for your reply. Is. shard() partitioned only when generating the load_dataset() object [image] But this error occurred during load_dataset() [image] [image]

It may also be another limitation of PyArrow. If you set num_shards to around 20, maybe it will work… I hope it does. [Data] Overflowing Pyarrow buffer when using `read_binary` to read files > 2Gb opened 09:42PM - 29 Oct 24 UTC (UTC) …

I have already set num_stards to 100, but the same error still exists data_set = load_dataset(self.data_file_path, cache_dir=cache_dir, split=“train”) data_set = data_set.shard(num_shards=100, index=0) It seems that this error already exists when executing load_dataset() [image]

LoadDataSet pyarrow.lib.ArrowCapacityError

🤗Datasets

John6666 January 12, 2025, 5:18am 4

How about trying .shard()?

Topic		Replies	Views
Strange pyarrow error when extracting rows from a public dataset Intermediate	2	365	April 30, 2025
Unable to Load Dataset Using `load_dataset` 🤗Datasets	10	1080	March 11, 2025
Proprietary database load error: TypeError: Argument 'storage' has incorrect type (expected pyarrow.lib.Array, got pyarrow.lib.ChunkedArray) 🤗Datasets	2	1183	January 25, 2022
Error converting np float32 🤗Datasets	3	87	August 5, 2025
Exceeded maximum rows when load_dataset for JSON 🤗Datasets	4	1187	April 6, 2023

LoadDataSet pyarrow.lib.ArrowCapacityError

Related topics