I am trying to push a large dataset with help of dataset.push_to_hub() While pushing, it gives HTTP error :504 requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/name/repo-name/upload/main/data/train2-00041-of-00064.parquet How to avoid…

Hi ! As discussed on GitHub, feel free to try again (the server might have had some issues). We’re also adding a retry mechanism to workaround 504 errors: Retry HfApi call inside push_to_hub when 504 error by albertvillanova · Pull Request #3886 · huggingface/datasets · GitHub We’ll do a new relea…

I’ve been getting this today (200GB dataset, push in python), just love to raise this is still a problem (and it’d be so great to be able to resume pushes since they can take a really long time) Creating parquet from Arrow format: 100%|██████████████████████████████████████████████████| 48/48 [00:0…

Possibly .push_to_hub() issue? @lhoestq [image] 504 Gateway Time-out when uploading large dataset to enterprise organization 🤗Datasets Hi We recently bought an enterprise subscription in order to increase our data usage limits. However, it is no longer possible for me to…

I’m considering writing code to build and save the large dataset to local hdd Try this to upload it without the fails Upload files to the Hub Good idea?

(note I’m calling push_to_hub on a DatasetDict, to an enterprise organization)

Good idea? Yea. In that case, you could use this. [image] HfApi Client We’re on a journey to advance and democratize artificial intelligence through open source and open science.

cc @Wauplin maybe a transient 504 issue ? Anyway @davidhhmack making push_to_hub easier to resume is definitely something we’ll look into !

Thanks, I appreciate it! It’s crushing to get hours through an upload, then have it die

HTTP 504: Gateway timeout error when pushing dataset

🤗Datasets

John6666 March 3, 2025, 4:16pm 4

Possibly .push_to_hub() issue? @lhoestq

Topic		Replies	Views
504 Gateway Time-out when uploading large dataset to enterprise organization 🤗Datasets	3	156	March 3, 2025
huggingface_hub.errors.HfHubHTTPError: Spaces	1	262	January 17, 2025
Failed to commit 504 Server Error Gateway Time-out for url Beginners	1	126	December 26, 2024
Gateway timeout on huggingface datasets 🤗Hub	5	152	September 23, 2025
HTTP 400 on push_to_hub with datasets 🤗Hub	2	1152	October 11, 2022

HTTP 504: Gateway timeout error when pushing dataset

Related topics