Upload a large folder from S3 to a dataset

DinoDS
Wed, 04 Mar 2026 19:49:50 +0000

upload_large_folder() needs a local filesystem path (a str / PathLike). An S3File handle is a file-like object, not a directory path, so you hit TypeError: expected str ... not S3File.

With Zarr, the practical workarounds are:

Option 1: Sync S3 → local dir, then upload the local dir

This is the simplest because Zarr is a directory tree.

from pathlib import Path
import subprocess
from huggingface_hub import upload_large_folder

destination_dataset = "stefanches/idr0012-fuchs-cellmorph-S-BIAD845"
s3_uri = "s3://bia-integrator-data/S-BIAD845/.../009bd3ab...zarr"
local_dir = Path("/tmp/009bd3ab...zarr")

# 1) download folder
subprocess.run(["aws", "s3", "sync", s3_uri, str(local_dir)], check=True)

# 2) upload folder
upload_large_folder(
    folder_path=str(local_dir),
    repo_id=destination_dataset,
    repo_type="dataset",
    path_in_repo="S-BIAD845/009bd3ab...zarr",  # optional, keeps structure
)

Why this works: you give HF a real directory path.

Option 2: Use snapshot_download style pattern in reverse by cloning the dataset repo + git lfs

For very large folder trees, you can often do better by pushing via git-lfs after cloning:

1. git lfs install

2. git clone https://huggingface.co/datasets/<repo>

3. copy the .zarr/ folder into the repo

4. git lfs track "*.zarr/**" (or track large file patterns)

5. git add/commit/push

This can be more robust than API upload for huge trees.

Option 3: Package the Zarr into a tar/zip, upload one object

If you just need storage and later download, bundle the folder:

tar -cf zarr.tar your.zarr/ (optionally compress)

upload the tar as a single file

Tradeoff: users must extract before opening as Zarr.

Why the 128 files/min limit bites Zarr

Zarr stores arrays as many chunk files, so you are effectively uploading thousands of small objects. Any per-file rate limit will hurt.