Aya-101-GGUF

This repo contains GGUF format model files for Cohere's Aya-101 model

Quantized using Huggingface's candle framework

How to use with Candle's quantized T5 example

Visit the candle T5 example for more detailed instruction

1. Clone candle repo:

git clone https://github.com/huggingface/candle.git
cd candle/candle-examples

2. Run the following command:

cargo run --example quantized-t5 --release  -- \
  --model-id "kcoopermiller/aya-101-GGUF" \
  --weight-file "aya-101.Q2_K.gguf" \
  --config-file "config.json" \
  --prompt "भारत में इतनी सारी भाषाएँ क्यों हैं?" \
  --temperature 0

Available weight files:

• aya-101.Q2_K.gguf
• aya-101.Q3_K.gguf
• aya-101.Q4_0.gguf
• aya-101.Q4_1.gguf
• aya-101.Q4_K.gguf
• aya-101.Q5_0.gguf
• aya-101.Q5_1.gguf
• aya-101.Q5_K.gguf
• aya-101.Q6_K.gguf
• aya-101.Q8_0.gguf
• aya-101.Q8_1.gguf (not supported on candle yet)
• aya-101.Q8_K.gguf (not supported on candle yet)