Instructions to use mjbommar/magic-bert-tokenizer-8k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mjbommar/magic-bert-tokenizer-8k with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("mjbommar/magic-bert-tokenizer-8k", device_map="auto") - Notebooks
- Google Colab
- Kaggle
| { | |
| "bos_token": "<|start|>", | |
| "cls_token": "<|cls|>", | |
| "eos_token": "<|end|>", | |
| "mask_token": "<|mask|>", | |
| "pad_token": "<|pad|>", | |
| "sep_token": "<|sep|>", | |
| "unk_token": "<|unk|>" | |
| } | |