Instructions to use barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit with MLX:

# Make sure mlx-vlm is installed
# pip install --upgrade mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model, processor = load("barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit")
config = load_config("barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit")

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

# Generate output
output = generate(model, processor, formatted_prompt, image)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent

How to use barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit

MTPLX 3-bit conversion of barozp/Qwen3.8-27B-Opus-Distill-v2 — the bugfix release of Qwen3.8-27B-Opus-Distill ("v1") — for speculative decoding on Apple Silicon via mtplx.

Sibling formats: the bf16 safetensors source above, the GGUF collection for llama.cpp/Ollama/LM Studio, and the 4-bit MTPLX repo.

Converted straight from the bf16 safetensors weights (not from any GGUF quant), so there is no dequantize-requantize drift in the chain.

Why this release exists

A community member (zxbc2023) reported a clean, deterministic bug on v1: certain stacked output-format constraints (e.g. "no prose" + "no markdown") sent the model into a non-converging self-verification loop, burning the entire token budget with zero visible output. Root cause traced to part of v1's training data being sourced from a dataset with reconstructed (not verbatim) reasoning traces. This release retrains on a rebuilt dataset — every row traced to a verified genuine Opus source, every row scanned (30-way parallel LLM-judge pass over the full set) and dropped if it showed the same pathological pattern.

MTPLX 3-bit keeps that fix intact and adds self-speculative decoding: the bf16 MTP head proposes draft tokens, the quantized trunk verifies them, all in a single MLX process. At ~13.6 GB on disk it fits comfortably on 24 GB unified-memory Macs while preserving vision.

Bug fix validation

Re-ran the reporter's exact repro (temp=0, seed=1234, fib(n) with stacked constraints):

Test	v1	v2
"no prose" + "no markdown"	3000/3000 tokens, 0 visible output	87/4096 tokens, clean
"no prose" + "no comments"	3000/3000 tokens, 0 visible output	73/4096 tokens, clean
Generalization (different task)	--	43/4096 tokens, clean

All three converge quickly, </think> closes normally — no loop.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (--limit 500). Same protocol as v1, so the delta column is the meaningful signal.

Task	Metric	Base	v2	Delta
wikitext	word perplexity (lower better)	8.4335	8.3788	-0.055
mmlu	acc	0.8494	0.8476	-0.002
hellaswag	acc_norm	0.7420	0.7500	+0.008
arc_challenge	acc_norm	0.5880	0.6220	+0.034
gpqa_diamond	acc_norm	0.2323	0.4697	+0.237

General knowledge (MMLU) and language modeling (wikitext) are essentially unaffected — in the same noise band as v1. Reasoning (ARC, GPQA) shows a large, real gain over base, not comparable to Qwen's own published GPQA number (thinking-mode-on, different harness) — this is a same-protocol base-vs-distill delta.

Available MTPLX conversions

Repo	Bits	Size (on disk)	Use case
this repo (3-bit)	3	~13.6 GB (12.6 GiB)	tight unified memory, recommended entry point
barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-4bit	4	~16.9 GB	best quality/size balance, highest speedups

K/MT comparison for context: MLX-4bit (14.97 GB, no MTP) and MLX-8bit (27.49 GB) exist for non-speculative MLX workflows; GGUF covers llama.cpp.

Which one to pick:

Best speculative speedup + quality headroom → 4-bit MTPLX (2.39x at D3 in our tests)
Tightest fit / 24 GB Mac → this 3-bit MTPLX (1.63x at D2, still strong)
No speculation needed → MLX-4bit; near-lossless → MLX-8bit / GGUF Q6_K / Q8_0

Conversion notes

Source: barozp/Qwen3.8-27B-Opus-Distill-v2 (bf16, barozp--Qwen3.8-27B-Opus-Distill-v2 local cache), forge-local via mtplx==2.9.1
Recipe: body_bits=3, body_group_size=64, body_mode=affine, mtp_policy=keep_bf16, quantized trunk + bf16 MTP sidecar
MTP contract: base_hidden_variant=post_norm, hidden_variant=post_norm, concat_order=embedding_hidden, mtp_position_mode=local, mtp_quant_group_size=64, mtp_quant_mode=affine
Output: 3 safetensors shards + model-vision.safetensors (879 MB, 333 tensors, bf16) + mtp.safetensors (810 MB, bf16 sidecar) + model.safetensors.index.json; architecture qwen3_next (dense 27B, 64 layers, hybrid Gated-DeltaNet / full-attention)
Forged at: 2026-08-24T10:32:21+03:00 on Apple M5 Pro (18 CPU / 20 GPU, 24 GB unified memory) — tuned for 24 GB Macs · macOS 27.0 arm64, mtplx_runtime.json ships in-repo as provenance

Vision

Unlike the GGUF builds (where the tower ships separately as mmproj), the vision encoder is inside these weights — image input works out of the box, no extra file to load. The tower is stored separately as model-vision.safetensors but is auto-loaded by mtplx; you just pass --image:

mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit \
  --prompt "Describe this image." --image photo.jpg --depth 2

Text-only usage needs nothing extra. High-resolution photos expand into many visual tokens (~12k observed); cap resolution when needed (MTPLX inherits the max-pixels handling from the base processor).

MTPLX usage

# install
pip install -U mtplx

# text-only chat (verified optimum is --depth 2 for this 3-bit)
mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit \
  --prompt "Explain the Monty Hall problem briefly." \
  --depth 2 --max-tokens 512

# multimodal
mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit \
  --prompt "Describe this image." --image photo.jpg --depth 2

# OpenAI-compatible local server
mtplx serve --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit \
  --depth 2 --port 8080
# then: curl http://localhost:8080/v1/chat/completions ...

Defaults verified for this checkpoint: temperature=0.6, top_p=0.95, top_k=20, recommended_profile=sustained. Depth is the number of draft tokens proposed per step.

Verification results

Verified locally with mtplx forge verify (MLX backend, 512-token budget, quality gate enabled). Depth 0 is plain AR.

Depth	tok/s	vs AR	Acceptance by position	Quality
0 (AR)	21.4	1.00x	—	pass
1	32.5	1.52x	88.8%	pass
2	34.8	1.63x	83.7% / 65.1%	pass
3	26.3	1.23x	69.6% / 42.7% / 26.9%	pass

Recommended depth: 2 (mtp_depth_wins at D2; D3 passes quality but loses throughput)
Verdict: mtp_depth_wins; quality_rejected=[], failure_reasons=[]
Hardware: Apple M5 Pro (18 CPU / 20 GPU, 24 GB) · macOS 27.0 arm64 (Apple Silicon, MLX), mtplx 2.9.1, artifact sha256:2d5b4a0c...

All depths passed the quality gate; throughput is the tiebreaker. For a faster but slightly larger alternative, the 4-bit sibling sustains D3 at 2.39x on the same hardware.

Training details (source safetensors model)

Base: Qwen/Qwen3.8-27B — dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
Data: barozp/opus-reasoning-distill-v2 (11,716 examples, 100% verified-genuine Opus traces)
Run: 1 epoch (696 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~4h15m on A100 80GB
Final validation loss: 0.4334
Vision + MTP: carried over byte-for-byte from the base checkpoint — never trained