Instructions to use unsloth/Qwen3.6-35B-A3B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/Qwen3.6-35B-A3B-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="unsloth/Qwen3.6-35B-A3B-GGUF")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("unsloth/Qwen3.6-35B-A3B-GGUF", dtype="auto")

llama-cpp-python

How to use unsloth/Qwen3.6-35B-A3B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="unsloth/Qwen3.6-35B-A3B-GGUF",
	filename="BF16/Qwen3.6-35B-A3B-BF16-00001-of-00002.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": [
				{
					"type": "text",
					"text": "Describe this image in one sentence."
				},
				{
					"type": "image_url",
					"image_url": {
						"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
					}
				}
			]
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use unsloth/Qwen3.6-35B-A3B-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Use Docker

docker model run hf.co/unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

LM Studio
Jan

vLLM

How to use unsloth/Qwen3.6-35B-A3B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/Qwen3.6-35B-A3B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3.6-35B-A3B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

SGLang

How to use unsloth/Qwen3.6-35B-A3B-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/Qwen3.6-35B-A3B-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3.6-35B-A3B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/Qwen3.6-35B-A3B-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3.6-35B-A3B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Ollama
How to use unsloth/Qwen3.6-35B-A3B-GGUF with Ollama:
```
ollama run hf.co/unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
```

Unsloth Studio new

How to use unsloth/Qwen3.6-35B-A3B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3.6-35B-A3B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3.6-35B-A3B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/Qwen3.6-35B-A3B-GGUF to start chatting

Pi new

How to use unsloth/Qwen3.6-35B-A3B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use unsloth/Qwen3.6-35B-A3B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use unsloth/Qwen3.6-35B-A3B-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
```

Lemonade

How to use unsloth/Qwen3.6-35B-A3B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.6-35B-A3B-GGUF-UD-Q4_K_M

List all available models

lemonade list

Qwen3.6 GGUF Benchmarks

#10

pinned

by danielhanchen - opened Apr 17

Discussion

danielhanchen

Unsloth AI org Apr 17

•

edited Apr 19

Hey guys, we ran Qwen3.6-35-A3B GGUF performance benchmarks to help you choose the best quant for the size.

Unsloth ranks first in 21 of 22 model sizes on mean KL divergence, making them SOTA.

Benchmarks + Guide overview: https://unsloth.ai/docs/models/qwen3.6#unsloth-gguf-benchmarks

danielhanchen pinned discussion Apr 17

floory

Apr 17

This comment has been hidden (marked as Off-Topic)

danielhanchen

Unsloth AI org Apr 17

The x axis is aligned properly, the Q3 kl quants are all bigger than 15gb.

I had to hide your comments due to unnecessary drama @floory

jerobnd

Apr 17

•

edited Apr 19

I've run benchmarks on the first 100 SWE-bench Verified samples using various Unsloth quantizations.

Model	tests	resolved	unresolved	error	incomplete
Qwen3.5-35B-A3B-Q4_K_M	100	59	25	14	2
Qwen3.5-35B-A3B-UD-Q6_K_XL	100	59	29	6	6
Qwen3.5-35B-A3B-Q8_0	100	59	30	8	3
Qwen3.5-122B-A10B-UD-Q5_K_XL	100	69	28	0	3
Qwen3.5-27B-UD-Q4_K_XL	100	71	26	2	1
Qwen3.6-35B-A3B-UD-Q8_K_XL	100	53	26	18	3

Errors: Output does not start with 'diff --git'. The model is failing t

Model	tests	resolved	unresolved	error	incomplete
Qwen3.5-35B-A3B-Q4_K_M	100	59	25	14	2
Qwen3.5-35B-A3B-UD-Q6_K_XL	100	59	29	6	6 o follow the system prompt.
Incomplete: It reached the 250-turn limit

I am utilizing mini-swe-agent with 250-turn limit and full context window. (Only 1 pass)

The benchmark for Qwen3.6-35B-A3B-UD-Q8_K_XL (Unsloth) was a disappointing surprise; it solved fewer tests and had more errors than Qwen3.5.

Has anyone else seen similar results?

shimmyshimmer

Unsloth AI org Apr 18

I've run benchmarks on the first 100 SWE-bench Verified samples using various Unsloth quantizations.

Model tests resolved unresolved error incomplete

Qwen3.5-35B-A3B-Q4_K_M 100 59 25 14 2

Qwen3.5-35B-A3B-UD-Q6_K_XL 100 59 29 5 5

Qwen3.5-35B-A3B-Q8_0 100 59 30 8 3

Qwen3.5-122B-A10B-UD-Q5_K_XL 100 69 28 0 3

Qwen3.5-27B-UD-Q4_K_XL 100 71 26 2 1

Qwen3.6-35B-A3B-UD-Q8_K_XL 100 53 26 18 3

Errors: Output does not start with 'diff --git'. The model is failing to follow the system prompt.
Incomplete: It reached the 250-pass limit

I am utilizing mini-swe-agent with a 250-pass limit and full context window.

The benchmark for Qwen3.6-35B-A3B-UD-Q8_K_XL (Unsloth) was a disappointing surprise; it solved fewer tests and had more errors than Qwen3.5.

Has anyone else seen similar results?

Hey thanks for the analysis. You're testing the first 100 SWE Bench results which isn't the best metric and a very low number of sample with no repeats. I'd recommend testing other quantizations as well with a larger sample size and more repeats

TimeLordRaps

Apr 18

This comment has been hidden (marked as Off-Topic)

TimeLordRaps

Apr 18

This comment has been hidden (marked as Off-Topic)

TimeLordRaps

Apr 18

This comment has been hidden (marked as Off-Topic)

TimeLordRaps

Apr 18

This comment has been hidden (marked as Off-Topic)

TimeLordRaps

Apr 18

This comment has been hidden (marked as Off-Topic)

TimeLordRaps

Apr 18

This comment has been hidden (marked as Off-Topic)

lyesrock

Apr 18

Whoa, TimeLord! I don't know what kind of carpet you're smoking, but I hope it's at least 4-bit quantized. 💨 You’re living in 2045 while we’re all just trying to reach the end of the prompt. Take a breath, the VRAM is safe!

TimeLordRaps

Apr 18

This comment has been hidden (marked as Off-Topic)

TimeLordRaps

Apr 18

This comment has been hidden (marked as Off-Topic)

TimeLordRaps

Apr 18

This comment has been hidden (marked as Off-Topic)

TimeLordRaps

Apr 18

This comment has been hidden (marked as Off-Topic)

jerobnd

Apr 18

I've run benchmarks on the first 100 SWE-bench Verified samples using various Unsloth quantizations.

Model tests resolved unresolved error incomplete

Qwen3.5-35B-A3B-Q4_K_M 100 59 25 14 2

Qwen3.5-35B-A3B-UD-Q6_K_XL 100 59 29 5 5

Qwen3.5-35B-A3B-Q8_0 100 59 30 8 3

Qwen3.5-122B-A10B-UD-Q5_K_XL 100 69 28 0 3

Qwen3.5-27B-UD-Q4_K_XL 100 71 26 2 1

Qwen3.6-35B-A3B-UD-Q8_K_XL 100 53 26 18 3

Errors: Output does not start with 'diff --git'. The model is failing to follow the system prompt.
Incomplete: It reached the 250-pass limit

I am utilizing mini-swe-agent with a 250-pass limit and full context window.

The benchmark for Qwen3.6-35B-A3B-UD-Q8_K_XL (Unsloth) was a disappointing surprise; it solved fewer tests and had more errors than Qwen3.5.

Has anyone else seen similar results?

Hey thanks for the analysis. You're testing the first 100 SWE Bench results which isn't the best metric and a very low number of sample with no repeats. I'd recommend testing other quantizations as well with a larger sample size and more repeats

Hi, Fixed an error in the description (Used mini-swe-agent with a maximum of 250 turns and a single pass).
Added benchmark results for Qwen3.6-35B-A3B-Q5_K_M by AesSedai, showing similar performance.

Model	tests	resolved	unresolved	error	incomplete
Qwen3.5-35B-A3B-Q4_K_M	100	59	25	14	2
Qwen3.5-35B-A3B-UD-Q6_K_XL	100	59	29	5	5
Qwen3.5-35B-A3B-Q8_0	100	59	30	8	3
Qwen3.5-122B-A10B-UD-Q5_K_XL	100	69	28	0	3
Qwen3.5-27B-UD-Q4_K_XL	100	71	26	2	1
Qwen3.6-35B-A3B-UD-Q8_K_XL	100	53	26	18	3
Qwen3.6-35B-A3B-Q5_K_M (AesSedai)	100	51	29	18	2

I am using the recomended parameters:
Thinking mode for precise coding tasks (e.g., WebDev):
temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

But I realized that Qwen used temp=1.0 and top_p=0.95 for SWE-bench. I don't know the reason for this difference.

curtis1969

Apr 18

•

edited Apr 24

the model for me produces way longer and improved outputs for the same prompt but it's more prone due to going over the top even the full precision one online I wonder how the other qwen 3.6 models are going to be update after 27b and the model is a major upgrade imo

vagrillo

Apr 18

There is an error in Qwen3.5-35B-A3B-UD-Q6_K_XL numbers

cduk

Apr 19

Hi, Fixed an error in the description (Used mini-swe-agent with a maximum of 250 turns and a single pass).
Added benchmark results for Qwen3.6-35B-A3B-Q5_K_M by AesSedai, showing similar performance.

Model tests resolved unresolved error incomplete

Qwen3.5-35B-A3B-Q4_K_M 100 59 25 14 2

Qwen3.5-35B-A3B-UD-Q6_K_XL 100 59 29 5 5

Qwen3.5-35B-A3B-Q8_0 100 59 30 8 3

Qwen3.5-122B-A10B-UD-Q5_K_XL 100 69 28 0 3

Qwen3.5-27B-UD-Q4_K_XL 100 71 26 2 1

Qwen3.6-35B-A3B-UD-Q8_K_XL 100 53 26 18 3

Qwen3.6-35B-A3B-Q5_K_M (AesSedai) 100 51 29 18 2

I am using the recomended parameters:
Thinking mode for precise coding tasks (e.g., WebDev):
temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

But I realized that Qwen used temp=1.0 and top_p=0.95 for SWE-bench. I don't know the reason for this difference.

Would you mind running also the Qwen3.5-9B-Q8_0 for comparison and/or can you give your run command to make comparable testing? Thanks.

LadyJun

Apr 19

I've run benchmarks on the first 100 SWE-bench Verified samples using various Unsloth quantizations.

Model tests resolved unresolved error incomplete

Qwen3.5-35B-A3B-Q4_K_M 100 59 25 14 2

Qwen3.5-35B-A3B-UD-Q6_K_XL 100 59 29 5 5

Qwen3.5-35B-A3B-Q8_0 100 59 30 8 3

Qwen3.5-122B-A10B-UD-Q5_K_XL 100 69 28 0 3

Qwen3.5-27B-UD-Q4_K_XL 100 71 26 2 1

Qwen3.6-35B-A3B-UD-Q8_K_XL 100 53 26 18 3

Errors: Output does not start with 'diff --git'. The model is failing to follow the system prompt.
Incomplete: It reached the 250-turn limit

I am utilizing mini-swe-agent with 250-turn limit and full context window. (Only 1 pass)

The benchmark for Qwen3.6-35B-A3B-UD-Q8_K_XL (Unsloth) was a disappointing surprise; it solved fewer tests and had more errors than Qwen3.5.

Has anyone else seen similar results?

Hi,

I also run my own tests to ensure I can rely on a model before using it, and can confirm that Qwen3.6 results are underwhelming compared to those of Qwen3.5.

I use the HumanEval and HumanEval+ test suite (164/164 tests, pass@1.)
Qwen3.6 35B-A3B UD-Q6_K_XL: 93.29% / 90.24%
Qwen3.5 35B-A3B UD-Q6_K_XL: 98.78%/93.9%

In my recap, Qwen3.6 is currently ranked #6, #1 being Gemma 4 31B dense with 100%/94.51% and #2 Gemma 4 26B-A4B sparse with 99.39% / 93.9%.

One positive thing about Qwen3.6 is reliable tool calling. Never experienced a single failed one so far. So I guess that for running whatever non-coding agentic tasks it could be considered better than its predecessor.

danielhanchen

Unsloth AI org Apr 19

Hi, Fixed an error in the description (Used mini-swe-agent with a maximum of 250 turns and a single pass).
Added benchmark results for Qwen3.6-35B-A3B-Q5_K_M by AesSedai, showing similar performance.

Model tests resolved unresolved error incomplete

Qwen3.5-35B-A3B-Q4_K_M 100 59 25 14 2

Qwen3.5-35B-A3B-UD-Q6_K_XL 100 59 29 5 5

Qwen3.5-35B-A3B-Q8_0 100 59 30 8 3

Qwen3.5-122B-A10B-UD-Q5_K_XL 100 69 28 0 3

Qwen3.5-27B-UD-Q4_K_XL 100 71 26 2 1

Qwen3.6-35B-A3B-UD-Q8_K_XL 100 53 26 18 3

Qwen3.6-35B-A3B-Q5_K_M (AesSedai) 100 51 29 18 2

I am using the recomended parameters:
Thinking mode for precise coding tasks (e.g., WebDev):
temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

But I realized that Qwen used temp=1.0 and top_p=0.95 for SWE-bench. I don't know the reason for this difference.

Very interesting results thanks for sharing

islameissa

Apr 19

•

edited Apr 19

I've run benchmarks on the first 100 SWE-bench Verified samples using various Unsloth quantizations.

Model tests resolved unresolved error incomplete

Qwen3.5-35B-A3B-Q4_K_M 100 59 25 14 2

Qwen3.5-35B-A3B-UD-Q6_K_XL 100 59 29 5 5

Qwen3.5-35B-A3B-Q8_0 100 59 30 8 3

Qwen3.5-122B-A10B-UD-Q5_K_XL 100 69 28 0 3

Qwen3.5-27B-UD-Q4_K_XL 100 71 26 2 1

Qwen3.6-35B-A3B-UD-Q8_K_XL 100 53 26 18 3

Errors: Output does not start with 'diff --git'. The model is failing to follow the system prompt.
Incomplete: It reached the 250-turn limit

I am utilizing mini-swe-agent with 250-turn limit and full context window. (Only 1 pass)

The benchmark for Qwen3.6-35B-A3B-UD-Q8_K_XL (Unsloth) was a disappointing surprise; it solved fewer tests and had more errors than Qwen3.5.

Has anyone else seen similar results?

Can I ask how many times did you ran these tests.
Would you mind running it on Qwen3.5-27B-IQ4_XS is it like 15% faster on my GPU.

I also ran some specific bench markings that are related to my day to day work that I will post here in

islameissa

Apr 19

This is my Benchmarking based on tasks I do from day to day (real life)
I would appreciate your suggestions to improve this in anyway

Clinical AI Benchmark — LLM Evaluation Report

Benchmark date: April 2026
Judge model: GPT-5.4 (OpenAI Responses API) with manual validation.
Total scored rows: 10 runs × 18 prompts per model (local models); 1 run for quant variants
Evaluation domain: Clinical documentation, medical reporting, code generation, agentic workflows

Benchmark Design

18 prompts across 4 task types, each scored 0–100 by GPT-5.4 using a per-prompt rubric and a gold-standard reference note. All local models run via llama.cpp (/v1/chat/completions).

Task	Prompts	Description
Medical Notes	5	Clinical documentation: follow-up notes, consult letters, procedure records, progress notes, referrals — scored on factual accuracy, completeness, and absence of hallucination
Sleep Study	5	Structured HST interpretation reports — scored on correct AHI classification, treatment recommendation, and numerical fidelity
Code Generation	5	Real application code tasks (C++, Python, JavaScript) — scored on correctness, completeness, and adherence to requirements
Agentic	3	Multi-step reasoning workflows (clinic follow-up, screening research, file navigation) — scored on task completion and reasoning quality

Scores below 70 are flagged. A score of 0 is only assigned when the output is completely non-functional (refusal, off-topic, or structurally broken).

Main Leaderboard

10 runs for all primary models. Higher is better. Stdev reflects prompt-level variance across all scored rows.

Rank	Model	Quant	Overall ↓	± Stdev	Medical	Sleep	Code	Agentic	Runs
1	GPT-5.4 (OpenAI)	—	87.4	10.1	75.4	90.8	93.0	92.7	1
2	Qwen3.5-27B	UD_Q4_K_XL	77.0	14.8	62.0	83.2	82.7	82.0	10
3	Qwen3.6-35B	—	74.5	15.0	59.1	81.5	81.7	76.6	10
4	Gemma-4-26B	—	74.5	16.6	57.1	77.0	85.3	81.1	10
5	Qwen3.5-27B	IQ4_XS	73.6	14.3	65.8	81.8	66.6	82.3	1
6	Qwen3.5-35B	—	72.0	15.4	56.8	80.2	78.7	73.0	10
7	Ministral-3-14B	—	60.2	21.9	34.3	63.8	76.2	70.9	10

⚠️ IQ4_XS result is from 2 run and carries higher variance. Treat as indicative only.

Thinking Mode Comparison

Thinking mode tested on Gemma-4-26B and Qwen3.5-27B (UD_Q4_K_XL) — 2 runs each, all 18 prompts.

Model	Mode	Overall	Medical	Sleep	Code	Agentic
Qwen3.5-27B	Standard	77.0	62.0	83.2	82.7	82.0
Qwen3.5-27B	Thinking	71.4	58.8	84.7	78.2	75.7
Gemma-4-26B	Standard	74.5	57.1	77.0	85.3	81.1
Gemma-4-26B	Thinking	70.0	53.1	77.0	88.4	81.7

Model Notes

Model	Parameters	Context	Quantization	Notes
GPT-5.4	—	—	—	Also used as judge; scores may reflect self-familiarity
Qwen3.5-27B	27B	32k	UD_Q4_K_XL	Best local model overall
Qwen3.5-27B	27B	32k	IQ4_XS	Single-run result
Qwen3.6-35B	35B	32k	—	Strong on sleep studies
Gemma-4-26B	26B	128k	—	Best local model for code
Qwen3.5-35B	35B	32k	—	Intermittent agentic failures
Ministral-3-14B	14B	128k	—	Not recommended for clinical tasks

Methodology

Judge: GPT-5.4 via OpenAI Responses API with reasoning.effort = medium
Rubric: Per-prompt rubric with a gold-standard reference note for medical and sleep tasks; task-level rubric for code and agentic tasks
Scoring: 0–100 integer; breakdown by criterion returned but only total used for ranking
Parallelism: All models scored concurrently per prompt via asyncio.gather; judge called sequentially per model output
Runs: 10 independent runs for primary models; results aggregated as mean ± population stdev
Flagging: Scores below 70 flagged; prompt-level variance > 15 points across models flagged for review
Infrastructure: llama.cpp server per model on dedicated ports; all local models on the same hardware for fair TPS comparison

jerobnd

Apr 19

There is an error in Qwen3.5-35B-A3B-UD-Q6_K_XL numbers

Yes. I updated the table.

Model	tests	resolved	unresolved	error	incomplete
Qwen3.5-35B-A3B-UD-Q6_K_XL	100	59	29	6	6

akierum

Apr 24

Best Qwen 3.5 / 3.6 / Gemma4 Benchmak in this video https://www.youtube.com/watch?v=ONQcX9s6_co&t=512s

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment