How to use jadechoghari/tokenizer-lib-mean with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("jadechoghari/tokenizer-lib-mean", device_map="auto")
https://huggingface.co/jadechoghari/tokenizer-lib-mean/resolve/main/special_tokens_map.json
hf download hf://jadechoghari/tokenizer-lib-mean/special_tokens_map.json
curl -L -o special_tokens_map.json https://huggingface.co/jadechoghari/tokenizer-lib-mean/resolve/main/special_tokens_map.json
