What to do when HuggingFace throws "Can't load tokenizer"

tunggad
Fri, 22 Dec 2023 22:39:32 +0000

I got into the reported error above while following the guide Causal language modeling, I tried ONLY step 3 in my notebook and it worked - I can load the model and do inference on it, on hub and local as well, that means the tokenizer was missing, trainer.push_to_hub() did not generate/push it to hub. Thank you!