Add llama.cpp PR #24260 build and run instructions

README.md

@@ -29,6 +29,81 @@ tags:
 </div>
 
 
+## Run these GGUFs with llama.cpp
+
+These are GGUF quants of North-Mini-Code-1.0. The model uses the `cohere2moe` architecture, which is not in a stock llama.cpp release yet. Until [llama.cpp PR #24260](https://github.com/ggml-org/llama.cpp/pull/24260) is merged, build llama.cpp from that PR branch to load these files. Once the PR lands in a release, these same GGUFs will run on stock llama.cpp with no re-download, because they already declare `general.architecture = cohere2moe`.
+
+### 1. Build llama.cpp from PR #24260
+
+```bash
+git clone https://github.com/ggml-org/llama.cpp
+cd llama.cpp
+git fetch origin pull/24260/head:cohere2-moe
+git checkout cohere2-moe
+
+# CUDA build. Drop -DGGML_CUDA=ON for a CPU only build.
+cmake -B build -DGGML_CUDA=ON
+cmake --build build --config Release -j
+```
+
+The binaries are written to `build/bin/` (`llama-cli`, `llama-server`, `llama-quantize`).
+
+### 2. Download a quant
+
+```bash
+pip install huggingface_hub
+
+hf download unsloth/North-Mini-Code-1.0-GGUF \
+  --include "North-Mini-Code-1.0-UD-Q4_K_XL.gguf" \
+  --local-dir North-Mini-Code-1.0-GGUF
+```
+
+Every quant here is a single file except `BF16/`, which is split into two shards. To use a split set, download the whole folder and point llama.cpp at the first shard (`...-00001-of-00002.gguf`); it loads the rest automatically.
+
+### 3. Run
+
+Interactive chat with llama-cli:
+
+```bash
+./build/bin/llama-cli \
+  --model North-Mini-Code-1.0-GGUF/North-Mini-Code-1.0-UD-Q4_K_XL.gguf \
+  --jinja \
+  --n-gpu-layers 99 \
+  --ctx-size 16384 \
+  --temp 1.0 --top-p 0.95 \
+  -p "Write a python program to check if a string is a palindrome."
+```
+
+OpenAI compatible server with llama-server:
+
+```bash
+./build/bin/llama-server \
+  --model North-Mini-Code-1.0-GGUF/North-Mini-Code-1.0-UD-Q4_K_XL.gguf \
+  --jinja \
+  --n-gpu-layers 99 \
+  --ctx-size 16384 \
+  --host 0.0.0.0 --port 8080
+```
+
+Then query it:
+
+```bash
+curl http://localhost:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "messages": [{"role": "user", "content": "Write a python program to check if a string is a palindrome."}],
+    "temperature": 1.0,
+    "top_p": 0.95
+  }'
+```
+
+Notes:
+
+- Pass `--jinja` so the model chat template, including tool calling, is applied.
+- Recommended sampling settings are `temperature=1.0` and `top_p=0.95`.
+- Set `--n-gpu-layers 99` to offload all layers to GPU, or lower it to fit your VRAM. Use `--ctx-size` to set the context window (the model supports up to 256K).
+- `imatrix_unsloth.gguf_file` is the importance matrix used to build these quants. It is not a model and is not loaded at runtime.
+
 # **Model Card for North Mini Code**
 
 ## **Model Summary**