Working on a translator, hoping to do fine-tuning with a utf-16 dataset so I can get all the French accents etc. Datasets load_dataset() doesn’t seem to like non-utf-8 Is there a way to specify or does it HAVE to be utf-8? If it has to be utf-8, any suggestions for special characters?

“got an unexpected keyword argument ‘encoding’” datasets==2.3.2 my line looks like this raw_dataset = load_dataset(“json”, data_files=“./train.json”, encoding=“utf-16”)

I’ve merged a PR that adds this param to the JSON builder. You can access this feature by installing datasets from main with pip install git+https://github.com/huggingface/datasets.git.

Thanks Mario! You sir are a gentleman and a scholar. :wink:

UTF-16 for datasets?

🤗Datasets

mariosasko June 19, 2023, 7:16pm 2

You can pass encoding="utf-16" to the load_dataset call.

Random utf-8 errors from dataset

Topic		Replies	Views
Datasets.load_datasets fails 🤗Datasets	12	1115	October 11, 2024
UniDecodeError: 'charmap' codec can't decode byte from Load_dataset Beginners	0	87	December 5, 2024
Random utf-8 errors from dataset Intermediate	10	3981	December 8, 2023
Cant create dataset with encoding 🤗Datasets	1	755	November 26, 2023
UnicodeDecodeError when loading Mulit Lingual text file 🤗Datasets	1	2408	April 7, 2022

UTF-16 for datasets?

Related topics