Pre - Train model with inputs_embeds

hilala
Tue, 04 Jul 2023 12:11:56 +0000

Hi,
I am using BertForPreTraining to pre-train a model from scratch, is there a way that the input for the model would be vectors to inputs_embeds instead of tokens (input_ids) , while still implementing masking tasks? How can I train a model on MLM without the option to use [MASK] token?
Thanks,

Read full topic