What to do when HuggingFace throws "Can't load tokenizer"

Areeb123
Thu, 16 Nov 2023 14:43:43 +0000

"I encountered a similar error but was able to resolve it by referring to the Hugging Face documentation.

1. Initially, access the Hugging Face hub via the notebook by executing the following commands:

!pip install huggingface_hub
from huggingface_hub import notebook_login
notebook_login()

Note: Two types of tokens, namely ‘read’ and ‘write’, are generated in your huggingface hub. The ‘write’ token should be utilized for authorization.

2. Begin by pushing the files associated with the model to the hub:

model.push_to_hub(“Model_Name”)

3. Similarly, push the tokenizer-related files to the hub:

tokenizer.push_to_hub(“Model_Name”)

And that’s it, your problem is solved