Add a subset to a dataset from CLI?

geoffnn
Wed, 05 Feb 2025 15:19:26 +0000

Hi!
I’m trying to modify a dataset on HF, that contains subsets – essentially processing it and re-uploading it. e.g. this one Salesforce/lotsa_data · Datasets at Hugging Face
What I’m doing currently is download the parquet for each subset, process it independently, store it in an S3 folder, and then use the CLI to upload the folder. This does not result in a dataset with subsets…
What’s the fastest way of reproducing the structure of the initial dataset after processing?

I think this is related to https://discuss.huggingface.co/t/create-multiple-dataset-subsets-at-the-same-time/12997
Thanks,

Read full topic