Hello everyone, I’m creating a HF dataset in an Azure VM by first reading a file from generator, doing some pre processing and then using save to disk. Im saving to disk on a blobfuse mount. In the middle of the save_to_disk operation I was running out of disk, and i think it is related to HF cach…

Hmm… @lhoestq [image] "No space left on device" when using HuggingFace + SageMaker Amazon SageMaker Hi, I’m trying to train a model using a HuggingFace estimator in SageMaker but I keep getting this error after a few minutes: [1,15]: File “pyarrow/ipc.pxi”, line 365, in …

I’ve dug into HF code based on the error message I receive below, and I think I have an idea of what’s going on: File “/opt/conda/envs/ptca/lib/python3.10/site-packages/datasets/io/generator.py”, line 49, in read self.builder.download_and_prepare( File “/opt/conda/envs/ptca/lib/python3.10/site-pa…

In a cloud environment, there are a lot of problems that can occur, such as the cache being deployed to a directory without permissions… However, if it was working until a short time ago, maybe it can be restored by downgrading the library? no space left on device …

BTW, in the Hugging Face-related libraries, loading and saving will behave differently depending on the version of the library. The following is an example of fixing it to a very old version (from October last year). pip install huggingface_hub==0.25.2 Client Challenge …

datasets uses shutil.disk_usage() to know if there is enough space before writing a (potentially huge) dataset (maybe if it says zero it can let the writing begin - it should fail anyway if it’s really zero under the hood ? that part of datasets is open to contributions btw if you want to improve i…

HF cache no space left on device

🤗Datasets

John6666 March 29, 2025, 8:09pm 2

Hmm… @lhoestq

Topic		Replies	Views
SageMaker OS Error No Space Left On Device while trying to train Falcon40B Amazon SageMaker	3	1402	August 24, 2023
"No space left on device" when using HuggingFace + SageMaker Amazon SageMaker	39	26495	October 10, 2023
"no space left on device" when downloading a large model for the Sagemaker training job Amazon SageMaker	4	5177	July 18, 2024
Load_dataset("file_location") disk quota out of space 🤗Datasets	1	1243	April 18, 2024
Cannot use Hugging Face cache on a read-only filesystem 🤗Transformers	3	657	January 29, 2025

HF cache no space left on device

Related topics