Hf upload-large-folder failed to commit

yirens
Fri, 02 Jan 2026 03:49:26 +0000

Hi All,

I am trying to upload a CFD dataset to huggingface. Due to the dataset nature, the data size is 1.1TB.

I was using upload-large-folder feature of the hf CLI for uploading. The process was normal until over 1TB files are already committed. After that, I always get the following error for hf CLI:
Failed to commit: The read operation timed out Failed to commit 20 files at once. Will retry with less files in next batch.

The hf upload-large-folder was able to hash and pre-upload, but simply cannot commit the pre-uploaded files to the dataset repository.

I wonder if the community has came across this issue, and if so what is the proper way to commit the files to dataset repo. Currently I am using

hf upload-large-folder <dataset_repo_name> . --repo-type=dataset

Any I tried to restart and resume uploading multiple times but the issue persist.

Also, here is the link to the CFD repo: yirens/double-delta-aero · Datasets at Hugging Face