# Using transformers (BERT, RoBERTa) without embedding layer **URL:** \ **Category:** Research\ **Created:** [December 13, 2020, 6:16pm UTC](https://discuss.huggingface.co/t/using-transformers-bert-roberta-without-embedding-layer/2807 "2020-12-13T18:16:24Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![tueboesen](https://avatars.discourse-cdn.com/v4/letter/t/8e7dd6/32.png) [@tueboesen](https://discuss.huggingface.co/u/tueboesen)\ **Post date:** [December 13, 2020, 6:16pm UTC](https://discuss.huggingface.co/t/using-transformers-bert-roberta-without-embedding-layer/2807/1 "2020-12-13T18:16:24Z")
I’m looking to train a RoBERTa model on protein sequences, which is in many ways similar to normal nlp training, but in others quite different. In the language of proteins, I have 20 characters instead of the normal 26 characters used in english (it is 26 right? :D), so that is rather similar. The big difference is that you don’t really combine the characters in proteins to actual words, but rather just keep each character as a distinct token or class. Hence essentially my input to the transformer model could just be a list of numbers ranging from 0-19. However that would mean that my input would only have 1 feature if I did that, and I’m not sure a transformer could work with that? I’m thinking of just doing a onehot encoding of these characters, which would give me 20 input features. However this is of course still very low in comparison to how normal transformers are trained, where d\_model is somewhere in the range of 128-512 if I understand correctly. Does anyone have any experience with anything like this? any good advice for how it is most likely to work? ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [December 13, 2020, 9:34pm UTC](https://discuss.huggingface.co/t/using-transformers-bert-roberta-without-embedding-layer/2807/2 "2020-12-13T21:34:43Z")
Hey, I’d recommend taking a look at this repo: [GitHub - agemagician/CodeTrans: Pretrained Language Models for Source code](https://github.com/agemagician/CodeTrans) by @agemagician . This repo uses transformer models for protein sequences if I understand it correctly. Also, taking a look at those models: > **[Rostlab (Rostlab)](https://huggingface.co/Rostlab)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. might help. Not sure if there is a notebook on doing protein sequence LM, maybe @agemagician has a good pointer by chance 🙂 ---
**Author:** ![agemagician](https://avatars.discourse-cdn.com/v4/letter/a/d07c76/32.png) [@agemagician](https://discuss.huggingface.co/u/agemagician)\ **Post date:** [December 13, 2020, 10:00pm UTC](https://discuss.huggingface.co/t/using-transformers-bert-roberta-without-embedding-layer/2807/3 "2020-12-13T22:00:24Z")
Hi @tueboesen, Yes, it will work. It can give you a very close results compared to MSA methods, sometimes even better results. If you combine it with MSA, it will even give you a better results compared to MSA methods alone. We have trained (Transformer XL, XLNet, Bert, Albert, Electra and T5) for Uniref100 and BFD dataset. I would recommend to simply use on of these models, because it requires tremendous amount of computing power to reach good results. You can find them here: > **[GitHub - agemagician/ProtTrans: ProtTrans is providing state of the art...](https://github.com/agemagician/ProtTrans)** > > ProtTrans is providing state of the art pretrained language models for proteins. ProtTrans was trained on thousands of GPUs from Summit and hundreds of Google TPUs using Transformers Models. - GitH... > **[Rostlab (Rostlab)](https://huggingface.co/Rostlab)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. You can find more details on our paper: > **[ProtTrans: Towards Cracking the Language of Life’s Code Through...](https://www.biorxiv.org/content/10.1101/2020.07.12.199554v2)** > > Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we... Facebook also trained Roberta using Unrief50 dataset: > **[GitHub - facebookresearch/esm: Evolutionary Scale Modeling (esm): Pretrained...](https://github.com/facebookresearch/esm)** > > Evolutionary Scale Modeling (esm): Pretrained language models for proteins - GitHub - facebookresearch/esm: Evolutionary Scale Modeling (esm): Pretrained language models for proteins Unfortunately, we don’t have a notebook for training from scratch, but you can find more details to replicate our results here: > > > Do you have the source code of the various pre-trained models? @patrickvonplaten : You meant : > **[GitHub - agemagician/ProtTrans: ProtTrans is providing state of the art...](https://github.com/agemagician/ProtTrans)** > > ProtTrans is providing state of the art pretrained language models for proteins. ProtTrans was trained on thousands of GPUs from Summit and hundreds of Google TPUs using Transformers Models. - GitH... Not : > **[GitHub - agemagician/CodeTrans: Pretrained Language Models for Source code](https://github.com/agemagician/CodeTrans)** > > Pretrained Language Models for Source code. Contribute to agemagician/CodeTrans development by creating an account on GitHub. 🙂 ProtTrans: Provides the SOT pre-trained models for protein sequences. CodeTrans: Provides the SOTpre-trained models for computer source code. ---
**Author:** ![tueboesen](https://avatars.discourse-cdn.com/v4/letter/t/8e7dd6/32.png) [@tueboesen](https://discuss.huggingface.co/u/tueboesen)\ **Post date:** [December 16, 2020, 3:35pm UTC](https://discuss.huggingface.co/t/using-transformers-bert-roberta-without-embedding-layer/2807/4 "2020-12-16T15:35:15Z")
> [@agemagician](#): > > gy and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontie Wow this is an amazing response, thank you so much for this. I will need some time to digest it all, but this is exactly what I need! ---
**Author:** ![tueboesen](https://avatars.discourse-cdn.com/v4/letter/t/8e7dd6/32.png) [@tueboesen](https://discuss.huggingface.co/u/tueboesen)\ **Post date:** [December 16, 2020, 4:21pm UTC](https://discuss.huggingface.co/t/using-transformers-bert-roberta-without-embedding-layer/2807/5 "2020-12-16T16:21:56Z")
Is there a way for me to use any of the models to return probability distributions? More specifically I would like to see how exactly the model has learned and test it out a bit. To this effect I would love to be able to feed it a protein sequence where I have masked out some of the amino acids, and then have it return a probability distribution for the full returned protein. I’m sure this is possible, after all this is how the model was trained in the first place, but I’m just a bit overwhelmed by all the models, so I haven’t managed to figure out how to do this. ---
**Author:** ![agemagician](https://avatars.discourse-cdn.com/v4/letter/a/d07c76/32.png) [@agemagician](https://discuss.huggingface.co/u/agemagician)\ **Post date:** [December 16, 2020, 5:37pm UTC](https://discuss.huggingface.co/t/using-transformers-bert-roberta-without-embedding-layer/2807/6 "2020-12-16T17:37:43Z")
You can find an answer to your question here: [https://github.com/agemagician/ProtTrans/issues/5](https://github.com/agemagician/ProtTrans/issues/5) ---
**Author:** ![tueboesen](https://avatars.discourse-cdn.com/v4/letter/t/8e7dd6/32.png) [@tueboesen](https://discuss.huggingface.co/u/tueboesen)\ **Post date:** [December 16, 2020, 7:19pm UTC](https://discuss.huggingface.co/t/using-transformers-bert-roberta-without-embedding-layer/2807/7 "2020-12-16T19:19:29Z")
Hmm that still doesn’t quite do it unless I’m missing something. This does allow masking of a sequence, but you can only mask 1 amino acid in the sequence, and it doesn’t give the actual probabilities on output, but only the top5 probabilities for that single masked amino acid. ---
**Author:** ![agemagician](https://avatars.discourse-cdn.com/v4/letter/a/d07c76/32.png) [@agemagician](https://discuss.huggingface.co/u/agemagician)\ **Post date:** [December 16, 2020, 7:55pm UTC](https://discuss.huggingface.co/t/using-transformers-bert-roberta-without-embedding-layer/2807/8 "2020-12-16T19:55:38Z")
You can send “top\_k” parameter to “fill-mask” method, to return more/all tokens. Check here: > If it is still doesn’t fit your use-case, then you have to implement it your self. ---
**Author:** ![agemagician](https://avatars.discourse-cdn.com/v4/letter/a/d07c76/32.png) [@agemagician](https://discuss.huggingface.co/u/agemagician)\ **Post date:** [December 16, 2020, 9:00pm UTC](https://discuss.huggingface.co/t/using-transformers-bert-roberta-without-embedding-layer/2807/9 "2020-12-16T21:00:52Z")
Something like that could be a good starting point for you: > **[Google Colaboratory](https://colab.research.google.com/drive/1E6oCfRcq895nySE0LdDOuoWm2bQ6MYhu?usp=sharing)**