Loading a dataset doesn't actually memory map

mariosasko
Mon, 04 Sep 2023 17:33:32 +0000

This is a complex issue - see https://github.com/huggingface/datasets/issues/4883.

PS: You can use import pyarrow as pa; pa.total_allocated_bytes() to check PyArrow’s RAM usage.

Read full topic