Upload a large folder from S3 to a dataset

stefanches
Wed, 15 Oct 2025 10:28:04 +0000

Hi everyone, trying to upload a Zarr image from S3 to HuggingFace. I have read Cloud storage, but I can not upload files one-by-one because the Zarr structure has many files for a single Zarr screen, and the Hugging Face upload terminates on 128 files / minute.

I’d like to upload the whole folder from s3 to HF:

from huggingface_hub import upload_large_folder

destination_dataset = "stefanches/idr0012-fuchs-cellmorph-S-BIAD845"

data_file = bia-integrator-data/S-BIAD845/009bd3ab-eb79-4cf4-8a11-ad028b827c03/009bd3ab-eb79-4cf4-8a11-ad028b827c03.zarr # really a directory in Zarr format

with s3.open(data_file) as zarr_path:

    path_in_repo = data_file[len(data_dir)-5:]

    upload_large_folder(

        folder_path=zarr_path,

        repo_id=destination_dataset,

        repo_type="dataset",

    )

    print(f"Uploaded {data_file} to {path_in_repo}")

However, I get the following error: TypeError: expected str, bytes or os.PathLike object, not S3File

What could be a possible workaround?