# RoFormer (Eng language) **URL:** \ **Category:** Models\ **Created:** [May 17, 2025, 11:49am UTC](https://discuss.huggingface.co/t/roformer-eng-language/155642 "2025-05-17T11:49:52Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![sourabhY](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sourabhy/32/40447_2.png) [@sourabhY](https://discuss.huggingface.co/u/sourabhY)\ **Post date:** [May 17, 2025, 11:49am UTC](https://discuss.huggingface.co/t/roformer-eng-language/155642/1 "2025-05-17T11:49:52Z")
is there a BERT/Roberta etc. any model with RoPE embedding on hugging face (for English language) or can some one provide any github repo/ help in making one from scratch . thanks in advance. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [May 17, 2025, 12:08pm UTC](https://discuss.huggingface.co/t/roformer-eng-language/155642/2 "2025-05-17T12:08:11Z")
Hmm… I don’t know anything about it, but it seems like there aren’t many English versions of ReFormer… > **[google/reformer-enwik8 · Hugging Face](https://huggingface.co/google/reformer-enwik8)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > [@Training a reformer from scratch](https://discuss.huggingface.co/t/training-a-reformer-from-scratch/8287): > > Hello, I want to train a reformer for a sequence classification task. The sequences are of protein so I thought of making a new tokenizer and then loaded as a reformer tokenizer which is defined as below. spm.SentencePieceTrainer.train(input='./sequences\_scope.txt', model\_prefix='REFORM', max\_sentence\_length=2000, vocab\_size=25) tokenizer = ReformerTokenizer("REFORM.model", padding=True) tokenizer.add\_special\_tokens({'pad\_token': '[PAD]'}) The dataset was created as below - with open("seque… > **[@Kseniase on Hugging Face: "8 types of RoPE As we always use Transformers,...](https://huggingface.co/posts/Kseniase/498106595218801)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > > > \### Feature request > > Hello, > > I would like if possible for Rotary Position Em…bedding scaling factors to be usable in the library. Currently this can only be done by monkey-patching the library. > > Namely, it requires modifying the: > \- \`max\_position\_embeddings\`: This can already be done via the model's config class or \`config.json\` > \- \`position\_scale\`: This variable doesn't exist currently, and there is no way to incorporate this effect at the moment without monkey-patching the existing \`LlamaRotaryEmbeddings\` class. (I'd also like to not step over toes of a possible future XPos implementation which also uses it's own scale for different purposes) > > \### Motivation > > Recently I demonstrated it is possible to drastically reduce training compute when fine-tuning pre-trained RoPE models with an adjusted scaling factor for the purpose of extending the context length of the model. This has the effect of interpolating the position embeddings making it easier to fine-tune the model using in-distribution positions as opposed to out-of-distribution positions typically used via pure extrapolation. There is an extended write-up with motivations here https://kaiokendev.github.io/context as well as the code I used (for the 8K example) can be found here https://huggingface.co/kaiokendev/superhot-13b-8k-no-rlhf-test/blob/main/llama\_rope\_scaled\_monkey\_patch.py > > Some existing discussions and benchmarks can be found here: https://github.com/ggerganov/llama.cpp/discussions/1965 > > Several models currently use this scaling feature, but they will not produce coherent output unless the scale is applied correctly during inference (scale is a hyperparameter): > \- https://huggingface.co/kaiokendev/superhot-30b-8k-no-rlhf-test > \- https://huggingface.co/Peeepy/Airoboros-13b-SuperHOT-8k > \- https://huggingface.co/emozilla/open\_llama\_7b-scaled > > EDIT: Meta has recently written a paper about it: https://arxiv.org/abs/2306.15595 > > \### Your contribution > > I would love to help in any way possible. While the basic implementation would be easy, I'm not sure what the best way could be for adding this modification (such as if users want to used a fixed scale versus having it dynamically applied based on the input sequence length) ---
**Author:** ![sourabhY](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sourabhy/32/40447_2.png) [@sourabhY](https://discuss.huggingface.co/u/sourabhY)\ **Post date:** [May 17, 2025, 12:22pm UTC](https://discuss.huggingface.co/t/roformer-eng-language/155642/3 "2025-05-17T12:22:30Z")
So if I want to create a nert model with rope from scratch is there any help. Any github repo or codes etc. Resources ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [May 17, 2025, 12:24pm UTC](https://discuss.huggingface.co/t/roformer-eng-language/155642/4 "2025-05-17T12:24:45Z")
Maybe ModernBERT has RoPE built in? > **[answerdotai/ModernBERT-base · Hugging Face](https://huggingface.co/answerdotai/ModernBERT-base)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[GitHub - AnswerDotAI/ModernBERT: Bringing BERT into modernity via both architecture...](https://github.com/AnswerDotAI/ModernBERT)** > > Bringing BERT into modernity via both architecture changes and scaling > **[GitHub - AaryanSahu/BERT-with-RoPE: Implementation of BERT from scratch(both...](https://github.com/AaryanSahu/BERT-with-RoPE)** > > Implementation of BERT from scratch(both pre-training and fine-tuning) using the RoPE embeddings. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [May 17, 2025, 12:26pm UTC](https://discuss.huggingface.co/t/roformer-eng-language/155642/5 "2025-05-17T12:26:40Z")
ModernBERT > **[Finally, a Replacement for BERT: Introducing ModernBERT](https://huggingface.co/blog/modernbert)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[ModernBERT - a answerdotai Collection](https://huggingface.co/collections/answerdotai/modernbert-67627ad707a4acbf33c41deb)** > > Bringing BERT into modernity via both architecture changes and scaling