What to do when HuggingFace throws "Can't load tokenizer"

Meera-Datey
Sun, 05 May 2024 07:33:03 +0000

EssamWisam:

remi/bertabs-finetuned-extractive-abstractive-summarization

You may encounter this error if the model is a gated model, and you need to accept the licensing terms before using the model.

Go to the Model Card in hugging face, accept the licensing term, generate key and add the key in your program

import os
os.environ['HF_TOKEN']='my-key'

this worked for me.