Hi, I have added custom tokens using this code: # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = AutoModelForMaskedLM.from_pretrained('bert-base-uncased') num_added_toks = tokenizer.add_tokens(['😎', '🤬'])…

hey @anon58275033 without seeing the full code it’s a bit hard to debug, but my first question would be whether you tokenized the corpus after adding the new tokens? if yes, did you observe whether the tokenization is working as expected?

Please stop spamming every topic with this other topic. If no one answered it, you should make sure you gave all the information inside for other people to properly help you.

hey @anon58275033 you can tokenize the corpus in the usual way after you’ve added new tokens with tokenizer.add_tokens. since it seems you’re doing masked language modeling, you might want to check out this tutorial to see how this is done: Google Colaboratory

Hi, I have checked out that tutorial. Is it possible to add the emojis to the vocabulary somehow?

Adding New Tokens - IndexError: index out of range in self

Beginners

anon58275033 June 11, 2021, 3:59pm 3

Hello @lewtun, how do I tokensize the corpus after adding the new tokens? Also, this issue in conjunction with this issue I am having: [HELP] How to include emojis in masked language modelling?

Topic		Replies	Views
Getting Error while adding new tokens in vocab Beginners	2	2726	June 19, 2022
How to add new tokens for existing masked language modelling? Beginners	3	723	June 11, 2021
NameError: name 'BertTokenizer' is not defined Beginners	2	8084	June 11, 2021
Adding a new mask_token for BERT-like models/tokenizers Intermediate	0	583	May 26, 2023
"IndexError: index out of range in self" for bert LM example on https://huggingface.co/transformers/quickstart.html Beginners	2	6402	October 29, 2020

Adding New Tokens - IndexError: index out of range in self

Related topics