Adding atomic / indivisible tokens to BPE tokenizer

[John6666]

Special tokens seem to be considered atomic. However, the implementation of special tokens is quite complex (it has been revised and changed over a long period of time), so it would be safer to search for information while working on it.

Are special_tokens the only tokens guaranteed to be atomic?

I’m building a BPE tokenizer from scratch, and I’d like to add some tokens to its vocabulary that are never broken apart and never included in merges. The second part of that (no tokens in the merges) is easy – the tokens don’t appear in the training text I give the tokenizer. But for the first part (never breaking my special token, e.g. [Y], into its [,Y, and ] components), I’m less sure of the correct process. I had though I could just add the tokens to the tokenizer using add_tokens() befo…

Correct way of adding special tokens before training a tokenizer