We have setup an aws ec2 instance and initialized https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct and mistralai/Pixtral-12B-2409 · Hugging Face models. when we create ami from these vm and create new instance, the transformers library isn’t able to load the pretrained models. When i delete .cache…

The symptoms are different but the problem could be similar to this. [image] "No space left on device" when using HuggingFace + SageMaker Amazon SageMaker Hi, I’m trying to train a model using a HuggingFace estimator in SageMaker but I keep getting this error after a few m…

Space doesn’t seem to be issue , I have enough free space on vm. The moment I delete .cache it redownloads and starts working.

Your error is coming from caching the dataset. Datasets is caching the dataset on disk to work with it properly. The default cache_dir is ~/.cache/huggingface/datasets. This directory seems not to be on the mounted EBS volume. I think it’s not that way, it’s around here.

I’m not trying to cache dataset, it’s just model weights and the ami has the weights in the root for itself. The primary purpose is to boot the process as fast as possible

Perhaps unresolved issue? Model cache does not work opened 05:46PM - 21 Feb 24 UTC (UTC) closed 08:04AM - 05 Apr 24 UTC (UTC) [bitcoinvsalts] bitcoinvsalts …

I have seen this issue in the GitHub issues I’m not facing this , but in my case it starts loading weights and remains stuck at 0% when I spin up new instance

Transformers cache not loading from a new vm

🤗Transformers

John6666 October 11, 2024, 8:00am 2

The symptoms are different but the problem could be similar to this.

Topic		Replies	Views
Load model from local cache directory in Sagemaker notebooks Amazon SageMaker	0	482	December 12, 2023
Cache large models on GPU instances between reboots Spaces	3	921	February 14, 2023
"No space left on device" when using HuggingFace + SageMaker Amazon SageMaker	39	26489	October 10, 2023
Load_cache before training 🤗Transformers	0	226	February 28, 2023
Huggingface Transformer code successfully gets executed on amazon web services but not on other server 🤗Transformers	1	1405	December 20, 2020

Transformers cache not loading from a new vm

Related topics