I am fine tuning custom dataset using this link https://huggingface.co/transformers/master/custom_datasets.html it is saving too many checkpoints that make my disk full, any suggestions? thanks

If it’s the part using Trainer, you can use the argument save_total_limit (in TrainingArguments) to limit the number of checkpoints kept.

thanks, indeed its very useful for me, One more question for my clarification, once it is saved, should I use the last checkpoint for using the model later? or my concept is not correct? Also I want to know how many checkpoints should we have? can we make it one checkpoint?

There is a load_best_model argument too, that will automatically load your best model (according to a metric you choose). It’s all in the docs :wink:

@sgugger How “smart” is this feature? I remember that in openNMT when you specify a max checkpoint this does not take into account the best evaluated checkpoint up to that point. In other words, if there is an old checkpoint that is the best, it can still get deleted. That should not be the case, so…

thanks for information, I am looking into the doc

thanks , it solved my issue, esp working with ’ load_best_model_at_end’

However for this load_best_model_at_end , does it mean the models saved during checkpoints, for example, I am using distillBert, the best model here refers to the checkpoint?

hi @BramVanroy , I hope you are well. sorry during training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell me how I can save the best model , my code is as follow, what I missed in t…

Checkpoints and disk storage

🤗Transformers

sgugger October 9, 2020, 1:06pm 9

It is smarter: the best checkpoint is always put at the top of the list of checkpoints, so it never gets deleted (or if it does, it’s a bug ).

Saving only the best performing checkpoint

Topic		Replies	Views
Saving only the best performing checkpoint 🤗Transformers	19	18463	May 23, 2023
Save only best model in Trainer 🤗Transformers	31	88355	June 25, 2024
Question Regarding trainer arguments:: load_best_model_at_end Beginners	2	2024	April 19, 2021
How to prevent too many checkpoints with run_clm.py? Beginners	5	1391	December 14, 2022
Checkpoint missing Optimizer.pt? How to Resume? 🤗Transformers	7	5698	May 18, 2021

Checkpoints and disk storage

Related topics