Upload a large folder from S3 to a dataset

pavan-ramkumar
Mon, 19 Jan 2026 18:13:48 +0000

I’m trying to make the second option work (not for zarr but for lance datasets). See: Support hosting lance / vortex / iceberg / zarr datasets on huggingface hub · Issue #7863 · huggingface/datasets · GitHub

However, path_or_fileobj=fh isn’t supported unless fh is one str, bytes, io.BufferedIOBase

```
ValueError: path_or_fileobj must be either an instance of str, bytes or io.BufferedIOBase. If you passed a file-like object, make sure it is in binary mode.
```

I’m not 100% sure but I think this effectively leads to downloading and uploading instead of direct streaming which can be a dealbreaker for large datasets (100 GB+)

Another issue is that Xet doesn’t support binary IO buffer, so it falls back to HTTP upload

```
Uploading files as a binary IO buffer is not supported by Xet Storage. Falling back to HTTP upload
```

Any pointers where to look? I’ve already looked at the HfFileSystem docs: Interact with the Hub through the Filesystem API