Add files using upload-large-folder tool

README.md

@@ -0,0 +1,106 @@
+---
+license: mit
+base_model: zai-org/GLM-5.3
+tags:
+- exl3
+- exllamav3
+- glm-5.3
+- dgx-spark
+- gb10
+- vllm
+- 1m-context
+- mtp
+library_name: vllm
+pipeline_tag: text-generation
+---
+
+# keys-GLM-5.3-EXL3 — full GLM-5.3 (753B MoE) as a 3-bit EXL3, quantized *for* a serving container
+
+**What this is:** the complete `zai-org/GLM-5.3` (78 layers, 256 routed experts, DeepSeek-style sparse MLA attention with indexer, one MTP block) with the **routed experts quantized to 3-bit EXL3 trellis** and everything else kept bf16/fp16. 308 GB in 41 safetensors shards. It serves on **four NVIDIA DGX Spark (GB10) with TP=4 + decode-context-parallel 4 at a 1,000,000-token context**, inside Mia AI Lab's vLLM container for the GLM-5.3-*Flash* EXL3 (`ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3`) plus the small patch set shipped in `serving/`.
+
+| | |
+|---|---|
+| Source | `zai-org/GLM-5.3` (FP8 block-scaled, 756 GB, 141 shards) |
+| Quantizer | exllamav3 **1.4.5** (`convert.py`, `-hb 16 -mb 16 -cpi 0`, calibration 250 rows × 2048) |
+| Routed experts | 57,600 matrices (75 MoE layers × 256 experts × gate/up/down) at **3.00 bpw**, `out_scales: always` |
+| Codebook | layer 3 = `mcg`, layers 4–77 = `mul1` (see *Codebook note*); the serving overlay handles both per layer |
+| Everything else | attention (q_a/q_b/kv_a/o_proj/indexer), dense MLP (layers 0–2), shared experts, router, norms, embed, lm_head: **bf16/fp16 (unquantized)** |
+| MTP block (layer 78) | **bf16**, kept — usable as an MTP speculator |
+| Per-rank footprint (TP=4) | ~75 GiB weights (+~5 GiB with the MTP block loaded) |
+| Context verified | needle-in-haystack PASS at 8.5K / 25K / 50K / 105K / 263K / **979K** tokens (DCP=4 sharded KV) |
+
+## Why we made it this way
+
+Every previous EXL3 GLM-5.3 we tried to serve on DGX Spark cost hours-to-days of patching: the published quants were made for TabbyAPI/exllamav3 (full-model EXL3, 2-bit heads, mul1 everywhere) and vLLM has no full-model EXL3 — Mia's container only implements EXL3 for the **routed experts**, with attention/dense/head native. So this time we quantized **to the container's ABI first**:
+
+* scope = routed experts only (the only thing the overlay's `Exl3Config` quantizes), uniform integer bits in the overlay's allowed set {3,4,5,6} → **3-bit**, because 4-bit experts (≈101 GB/rank) do not fit four 128 GB GB10s next to a 1M KV cache; 3-bit ≈ 78 GB/rank does.
+* `-hb 16 -mb 16`: head and MTP stay bf16 so the stock vLLM paths load them.
+* suffix layout `trellis/suh/svh/<codebook marker>` exactly as the overlay's loader expects, `quantization_config.json` with `quant_method=exl3, bits=3, scope=glm53_routed_experts_only`.
+
+It still needed a handful of patches (all in `serving/`), but they are small, mechanical, and now documented — the point of this repo.
+
+## How it was made (4× DGX Spark "encode farm")
+
+One 753B model does not fit a single 128 GB Spark for quantization, and exllamav3 has no multi-GPU convert. We built a rolling pipeline (`quant/orchestrate.py`):
+
+1. **Rolling source shards.** The FP8 source is 756 GB and not layer-ordered; the orchestrator keeps a 5-shard download window (sha-checked), runs `convert.py` for **two layers at a time** in a fresh container (`--max_module L+1`, then `-r` resume), and after each step rewrites finished shards **dropping only `model.layers.L.mlp.experts.*`** (norms/attention stay — the compile step needs them). Peak disk ≈ 660 GB; permanent shards = embed/lm_head, final norm, and the layer-78 MTP shard (must exist at every resume).
+2. **Remote expert-encoding farm** (`quant/exl3_farm.py` + `quant/patch_farm.py`). exllamav3's per-layer H (Hessian) capture is serial, but the 768 expert matrices per layer are independent once H is final. The patch adds remote work threads to `quantize_linears_parallel`: it ships each matrix + its finalized H over TCP (length-prefixed `torch.save`) to worker containers on the other three Sparks, which return `trellis/suh/svh/marker`. Verified bit-identical to local encoding. **One MoE layer: 49 min single-GPU → ~20–22 min on four.** Whole model: ~27 wall-clock hours including a power-cut restart.
+3. **Resume-safe.** A power interruption at 60/78 lost nothing: exllamav3 checkpoints after every layer; the in-flight layer is simply redone.
+4. **Compile.** `compile.py` needs `marisa_trie` (not in the base image) — `pip install marisa-trie` in the quant container.
+
+`quant/recipe.yaml` (58,167 tensor entries) is included; `quant/build.sh` + `quant/patch_aarch64.py` build exllamav3 1.4.5 on aarch64/CUDA 13 inside the Mia container image (AVX all-reduce/CPU-MoE stubs + pause intrinsic).
+
+### Codebook note (read before you quantize with exllamav3 ≥ 1.4)
+
+`convert.py -r` (resume) lets the **command-line default win over the stored args** for the codebook (`("codebook", True, "mul1")` in the arg table). We passed `-cb mcg` on the first run only; every resumed 2-layer run silently quantized with `mul1`. Result: layer 3 is `mcg`, layers 4–77 are `mul1`. Quality is identical (both are 3-bit trellis codebooks), but Mia's overlay was **mcg-only** — which is why `serving/exl3.py` now detects the codebook per layer and drives the kernels' `(mcg, mul1)` flag pairs. **If you resume an exllamav3 job, re-pass every quant flag.**
+
+## Serving on 4× DGX Spark (what `serving/` contains and why)
+
+Base: `ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3` + its `start-tp4.sh` launcher (github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks). Image layers we add (Dockerfiles included, build order `dcp → mul1 → v145 → keys → keys3`):
+
+| Patch | Why |
+|---|---|
+| `exl3.py` (overlay) | + `mul1` marker tensors and per-layer codebook detection; markers may be 0-d scalars; `NullConfig/InferParams` stub for exllamav3 ≥ 1.x's `LinearEXL3`; **MTP block exempt from EXL3** so `SPEC_METHOD=mtp` loads the bf16 MTP experts through the stock path |
+| exllamav3 **1.4.5** swap (`Dockerfile.v145`) | the image's exllamav3 0.0.43 `exl3_moe` kernel hard-rejects `mul1` (`TORCH_CHECK`); 1.4.5 has the same 29-arg signature (+`num_active`) and accepts it. Built against the same base torch, so the `.so` drops in |
+| `sparse_attn_indexer.py` top-k fallback (`Dockerfile.keys`) | prefill `persistent_topk` for top-k 2048 oversubscribes the GB10's 48 SMs; Mia's runtime patch only covers the decode ("kpool") path |
+| `sm120_dcp.py` + `mla_attn_dcp.py` (`Dockerfile.dcp`) | decode-context-parallel port of the SM120 sparse-MLA backend (KV sharded across TP ranks → 1M context on four GB10s; the stock stack replicates KV and tops out ~300K) |
+| `patch_ablit_dsv2.py` + `ablit_runtime.py` | optional load-time o_proj transplant hook for the full-model classes (`DeepseekV2Model`/`DeepSeekMTP`); off by default |
+| `patch_tp4_launcher.py` | adds `LOCAL_MODEL_DIR`/`LOCAL_DFLASH_DIR` (serve from an NFS dir) and `ABLIT_HOST_DIR` to `start-tp4.sh` |
+| `keys-launch.sh`, `env.tp4.example` | our launch wrapper (drops page caches on all ranks first) and the working `.env.tp4` |
+
+Launcher-level gotchas we hit (all in `env.tp4.example` comments): `--ulimit nofile=1048576` (NCCL "Too many open files" at 4 ranks), a rank whose fabric IP sits on the *second* CX7 port needs its own `WORKER_CX7_IF/IB`, the NFS server node must bind-mount its own export at the shared path, `gpu_memory_utilization` above ~0.82 is refused by vLLM's startup guard (the worker already holds ~14 GiB of CUDA/NCCL state at check time) — use `--kv-cache-memory-bytes` to size the KV cache explicitly.
+
+### Measured (4× GB10, TP=4, DCP=4, CUDA graphs, fp8 KV)
+
+| Config | C1 per-stream | C2 | C4 | C8 aggregate | TTFT @C8 |
+|---|---|---|---|---|---|
+| no speculation | 8–9 tok/s | 16 | 25 | 38 | ~2 s |
+| **MTP k=3** (bf16 MTP block) — prose | 10.4 | 15.9 | 21.9 | 31.5 | 2.1 s |
+| **MTP k=3** — code | 16.0 | 22.0 | 28.2 | 43.1 | 1.9 s |
+
+256-token outputs, thinking off. Prefill ≈ 320–410 tok/s (the limiter at depth: a 979K prompt takes ~52 min to first token; decode there is 12 tok/s). DFlash2 draft: loads, but the draft's sliding-window layers are rejected by vLLM's DCP path (`DCP not support sliding window`) — parked.
+
+## Quick start (4 Sparks)
+
+```bash
+git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks glm53-mia && cd glm53-mia
+python3 serving/patch_tp4_launcher.py start-tp4.sh          # LOCAL_MODEL_DIR + ABLIT_HOST_DIR knobs
+# build the image chain on EVERY rank (see serving/Dockerfile.* ; ~1 min each on top of the Mia base)
+cp serving/env.tp4.example .env.tp4                          # set rank IPs / NIC pins / LOCAL_MODEL_DIR=<this repo on shared storage>
+SKIP_PULL=1 SKIP_SHIP=1 SKIP_BUILD=1 SKIP_DOWNLOAD=1 SKIP_SYNC=1 SKIP_OVERLAY_VERIFY=1 ./start-tp4.sh
+```
+OpenAI-compatible API on `:8888`, served model name `GLM-5.3-EXL3`. Sampling: temperature 1.0, top_p 0.95, thinking on (GLM defaults).
+
+## Lessons for "quantize with the serving container in mind"
+
+1. Read the serving overlay's **loader** first: allowed scope, bits set, tensor suffixes, marker dtype/shape, codebook — and quantize to that ABI, not to the quantizer's defaults.
+2. Keep anything the container serves natively (attention, dense, head, MTP) unquantized; the memory cost is small next to the experts and it removes whole classes of patches.
+3. Verify tensor suffixes after the **first resumed** step, not at the end (see the codebook note).
+4. Budget memory per rank for weights + KV + the ~14 GiB the process holds before vLLM measures anything; size KV explicitly.
+5. A per-layer encode farm turns a multi-day single-GPU convert into an overnight job on the same boxes that will serve the model.
+
+## Credits
+
+Zhipu / zai-org for GLM-5.3 · turboderp for exllamav3 · Mia AI Lab for the GB10 EXL3 vLLM container and launcher · Blackfrost-Research for the DERISKED GLM-5.3 (source of the optional ablit donors; **not applied** in these weights) · Quantized and served by Kevin (@drowzeys) on a 4× DGX Spark cluster, 2026-09.
+
+License: the weights inherit the `zai-org/GLM-5.3` license (MIT). Patches in `serving/` are MIT; they modify Apache-2.0 vLLM/SGLang/exllamav3 code — see their headers.

chat_template.jinja

@@ -0,0 +1,251 @@
+[gMASK]<sop>
+{%- set effective_reasoning_effort = reasoning_effort if reasoning_effort is defined and reasoning_effort in ['low', 'high'] else 'max' -%}
+{%- if effective_reasoning_effort is not none -%}<|system|>Reasoning Effort: {{ effective_reasoning_effort | capitalize }}{%- endif -%}
+{%- set clear_thinking = clear_thinking if clear_thinking is defined else false -%}
+{%- if tools -%}
+{%- macro tool_to_json(tool) -%}
+    {%- set ns_tool = namespace(first=true) -%}
+    {{ '{' -}}
+    {%- for k, v in tool.items() -%}
+        {%- if k != 'defer_loading' and k != 'strict' -%}
+            {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%}
+            {%- set ns_tool.first = false -%}
+            "{{ k }}": {{ v | tojson(ensure_ascii=False) }}
+        {%- endif -%}
+    {%- endfor -%}
+    {{- '}' -}}
+{%- endmacro -%}
+{%- macro tool_references_to_response(refs) -%}
+    {{- '<tool_response><tools>\n' -}}
+    {%- for tr in refs -%}
+        {%- for tool in tools -%}
+            {%- if 'function' in tool -%}
+                {%- set tool = tool['function'] -%}
+            {%- endif -%}
+            {%- if tool.name == tr.name -%}
+                {{- tool_to_json(tool) + '\n' -}}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- endfor -%}
+    {{- '</tools></tool_response>' -}}
+{%- endmacro -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{%- if 'function' in tool -%}
+    {%- set tool = tool['function'] -%}
+{%- endif -%}
+{% if tool.defer_loading is not defined or not tool.defer_loading %}
+{{ tool_to_json(tool) }}
+{% endif %}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- elif item is mapping and item.type in ['image', 'image_url', 'video', 'video_url', 'audio', 'audio_url', 'input_audio'] -%}
+                {%- set media_type = item.type | replace('_url', '') | replace('input_', '') -%}
+                {{- "<reminder>You are unable to process this " ~ media_type ~ " because you don't have multi-modal input ability. Try different methods.</reminder>" }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- macro tool_response(text) -%}
+{{- '<tool_response>' + text + '</tool_response>' -}}
+{%- endmacro -%}
+{%- macro render_tool_response(m) -%}
+{%- if m.content is string -%}
+    {{- tool_response(m.content) -}}
+{%- elif m.content and m.content is not mapping and m.content.0.type == "tool_reference" -%}
+    {{- tool_references_to_response(m.content) -}}
+{%- elif is_list_of_outputs(m) -%}
+    {%- for tr in m.content -%}
+        {%- if tr.output is iterable and tr.output is not string and tr.output is not mapping and tr.output and tr.output.0.type == "tool_reference" -%}
+            {{- tool_references_to_response(tr.output) -}}
+        {%- else -%}
+            {{- tool_response(visible_text(tr.output)) -}}
+        {%- endif -%}
+    {%- endfor -%}
+{%- else -%}
+    {{- tool_response(visible_text(m.content)) -}}
+{%- endif -%}
+{%- endmacro -%}
+{%- macro id_of(obj) -%}
+    {%- if obj.tool_call_id -%}
+        {{- obj.tool_call_id -}}
+    {%- elif obj.id -%}
+        {{- obj.id -}}
+    {%- endif -%}
+{%- endmacro -%}
+{%- macro is_list_of_outputs(m) -%}
+    {%- if m.content and m.content.0.output is defined -%}1{%- endif -%}
+{%- endmacro -%}
+{%- macro has_dup_tool_result_id(lo, hi, target) -%}
+    {%- set ns_cnt = namespace(n=0) -%}
+    {%- for k in range(lo, hi + 1) -%}
+        {%- set m = messages[k] -%}
+        {%- if is_list_of_outputs(m) -%}
+            {%- for entry in m.content -%}
+                {%- if id_of(entry) == target -%}
+                    {%- set ns_cnt.n = ns_cnt.n + 1 -%}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- elif id_of(m) == target -%}
+            {%- set ns_cnt.n = ns_cnt.n + 1 -%}
+        {%- endif -%}
+        {%- if ns_cnt.n > 1 -%}{%- break -%}{%- endif -%}
+    {%- endfor -%}
+    {%- if ns_cnt.n > 1 -%}1{%- endif -%}
+{%- endmacro -%}
+{%- macro tc_id_exists(tcs, target) -%}
+    {%- set ns_f = namespace(found=false) -%}
+    {%- for tc in tcs -%}
+        {%- if id_of(tc) == target -%}
+            {%- set ns_f.found = true -%}
+            {%- break -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- if ns_f.found -%}1{%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) -%}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {%- set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{%- for m in messages -%}
+{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- elif '</think>' in content %}
+    {%- set reasoning_content = content.split('</think>')[0].split('<think>')[-1] %}
+    {%- set content = content.split('</think>')[-1] %}
+{%- endif %}
+{%- if (not clear_thinking or loop.index0 > ns.last_user_index) and reasoning_content is defined -%}
+{{ '<think>' + reasoning_content +  '</think>'}}
+{%- else -%}
+{{ '<think></think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{- '<tool_call>' + tc.name -}}
+{% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' -}}
+    {%- set block_start = loop.index0 -%}
+    {%- set ns_blk = namespace(end=block_start) -%}
+    {%- for j in range(block_start, messages|length) -%}
+        {%- if messages[j].role == 'tool' -%}
+            {%- set ns_blk.end = j -%}
+        {%- else -%}
+            {%- break -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- set ns_a = namespace(tool_calls=none) -%}
+    {%- if block_start > 0 and messages[block_start - 1].role == 'assistant' and messages[block_start - 1].tool_calls -%}
+        {%- set ns_a.tool_calls = messages[block_start - 1].tool_calls -%}
+    {%- endif -%}
+    {%- set ns_chk = namespace(can_sort=true) -%}
+    {%- if not ns_a.tool_calls -%}
+        {%- set ns_chk.can_sort = false -%}
+    {%- else -%}
+        {%- for k in range(block_start, ns_blk.end + 1) -%}
+            {%- set m = messages[k] -%}
+            {%- if is_list_of_outputs(m) -%}
+                {%- for entry in m.content -%}
+                    {%- set eid = id_of(entry) -%}
+                    {%- if not eid -%}
+                        {%- set ns_chk.can_sort = false -%}
+                    {%- elif has_dup_tool_result_id(block_start, ns_blk.end, eid) -%}
+                        {%- set ns_chk.can_sort = false -%}
+                    {%- elif not tc_id_exists(ns_a.tool_calls, eid) -%}
+                        {%- set ns_chk.can_sort = false -%}
+                    {%- endif -%}
+                {%- endfor -%}
+            {%- else -%}
+                {%- set tk_id = id_of(m) -%}
+                {%- if not tk_id -%}
+                    {%- set ns_chk.can_sort = false -%}
+                {%- elif has_dup_tool_result_id(block_start, ns_blk.end, tk_id) -%}
+                    {%- set ns_chk.can_sort = false -%}
+                {%- elif not tc_id_exists(ns_a.tool_calls, tk_id) -%}
+                    {%- set ns_chk.can_sort = false -%}
+                {%- endif -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {%- for i in range(ns_a.tool_calls | length) -%}
+            {%- set tc_id = id_of(ns_a.tool_calls[i]) -%}
+            {%- if not tc_id -%}
+                {%- set ns_chk.can_sort = false -%}
+            {%- endif -%}
+            {%- for j in range(i + 1, ns_a.tool_calls | length) -%}
+                {%- if id_of(ns_a.tool_calls[j]) == tc_id -%}
+                    {%- set ns_chk.can_sort = false -%}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endfor -%}
+    {%- endif -%}
+    {%- if ns_chk.can_sort -%}
+        {%- for tc in ns_a.tool_calls -%}
+            {%- set tc_id = id_of(tc) -%}
+            {%- for k in range(block_start, ns_blk.end + 1) -%}
+                {%- set m = messages[k] -%}
+                {%- if is_list_of_outputs(m) -%}
+                    {%- for entry in m.content -%}
+                        {%- set eid = id_of(entry) -%}
+                        {%- if eid == tc_id -%}
+                            {%- if entry.output is iterable and entry.output is not string and entry.output is not mapping and entry.output and entry.output.0.type == "tool_reference" -%}
+                                {{- tool_references_to_response(entry.output) -}}
+                            {%- else -%}
+                                {{- tool_response(visible_text(entry.output)) -}}
+                            {%- endif -%}
+                        {%- endif -%}
+                    {%- endfor -%}
+                {%- else -%}
+                    {%- set tk_id = id_of(m) -%}
+                    {%- if tk_id == tc_id -%}
+                        {{- render_tool_response(m) -}}
+                    {%- endif -%}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endfor -%}
+    {%- else -%}
+        {%- for k in range(block_start, ns_blk.end + 1) -%}
+            {{- render_tool_response(messages[k]) -}}
+        {%- endfor -%}
+    {%- endif -%}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '<think>' -}}
+{%- endif -%}

config.json

@@ -0,0 +1,792 @@
+{
+  "architectures": [
+    "GlmMoeDsaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "dtype": "bfloat16",
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "ep_size": 1,
+  "first_k_dense_replace": 3,
+  "head_dim": 192,
+  "hidden_act": "silu",
+  "hidden_size": 6144,
+  "index_head_dim": 128,
+  "index_n_heads": 32,
+  "index_share_for_mtp_iteration": true,
+  "index_skip_topk_offset": 3,
+  "index_topk": 2048,
+  "index_topk_freq": 4,
+  "index_topk_pattern": null,
+  "indexer_rope_interleave": true,
+  "indexer_types": [
+    "full",
+    "full",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared"
+  ],
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "kv_lora_rank": 512,
+  "max_position_embeddings": 1048576,
+  "mlp_layer_types": [
+    "dense",
+    "dense",
+    "dense",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse"
+  ],
+  "model_type": "glm_moe_dsa",
+  "moe_intermediate_size": 2048,
+  "moe_layer_freq": 1,
+  "moe_router_dtype": "float32",
+  "n_group": 1,
+  "n_routed_experts": 256,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 78,
+  "num_key_value_heads": 64,
+  "num_nextn_predict_layers": 1,
+  "pad_token_id": 154820,
+  "pretraining_tp": 1,
+  "q_lora_rank": 2048,
+  "qk_head_dim": 256,
+  "qk_nope_head_dim": 192,
+  "qk_rope_head_dim": 64,
+  "rms_norm_eps": 1e-05,
+  "rope_interleave": true,
+  "rope_parameters": {
+    "rope_theta": 8000000,
+    "rope_type": "default"
+  },
+  "routed_scaling_factor": 2.5,
+  "scoring_func": "sigmoid",
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "topk_method": "noaux_tc",
+  "transformers_version": "5.15.0",
+  "use_cache": true,
+  "v_head_dim": 256,
+  "vocab_size": 154880,
+  "quantization_config": {
+    "quant_method": "exl3",
+    "version": "1.4.5",
+    "bits": 3,
+    "head_bits": 16,
+    "calibration": {
+      "rows": 250,
+      "cols": 2048
+    },
+    "out_scales": "always",
+    "codebook": "mul1",
+    "original_quantization_config": {
+      "activation_scheme": "dynamic",
+      "fmt": "e4m3",
+      "quant_method": "fp8",
+      "weight_block_size": [
+        128,
+        128
+      ],
+      "modules_to_not_convert": [
+        "model.layers.59.post_attention_layernorm",
+        "model.layers.47.mlp.gate.e_score_correction_bias",
+        "model.layers.71.self_attn.kv_a_layernorm",
+        "model.layers.46.mlp.gate.e_score_correction_bias",
+        "model.layers.30.self_attn.kv_a_layernorm",
+        "model.layers.23.self_attn.kv_a_layernorm",
+        "model.layers.73.mlp.gate.e_score_correction_bias",
+        "model.layers.36.mlp.gate",
+        "model.layers.12.self_attn.q_a_layernorm",
+        "model.layers.51.mlp.gate",
+        "model.layers.47.self_attn.kv_a_layernorm",
+        "model.layers.45.self_attn.q_a_layernorm",
+        "model.layers.15.self_attn.q_a_layernorm",
+        "model.layers.39.input_layernorm",
+        "model.layers.50.self_attn.kv_a_layernorm",
+        "model.layers.19.post_attention_layernorm",
+        "model.layers.71.input_layernorm",
+        "model.layers.72.self_attn.kv_a_layernorm",
+        "model.layers.70.mlp.gate",
+        "model.layers.52.input_layernorm",
+        "model.layers.78.self_attn.kv_a_layernorm",
+        "model.layers.48.post_attention_layernorm",
+        "model.layers.4.self_attn.q_a_layernorm",
+        "model.layers.62.self_attn.q_a_layernorm",
+        "model.layers.4.post_attention_layernorm",
+        "model.layers.38.self_attn.indexers_proj",
+        "model.layers.61.mlp.gate.e_score_correction_bias",
+        "model.layers.14.mlp.gate.e_score_correction_bias",
+        "model.layers.36.self_attn.kv_a_layernorm",
+        "model.layers.65.mlp.gate",
+        "model.layers.13.self_attn.kv_a_layernorm",
+        "model.layers.57.mlp.gate",
+        "model.layers.65.self_attn.kv_a_layernorm",
+        "model.layers.68.input_layernorm",
+        "model.layers.70.self_attn.kv_a_layernorm",
+        "model.layers.24.mlp.gate.e_score_correction_bias",
+        "model.layers.4.input_layernorm",
+        "model.layers.20.input_layernorm",
+        "model.layers.65.self_attn.q_a_layernorm",
+        "model.layers.38.mlp.gate",
+        "model.layers.76.mlp.gate.e_score_correction_bias",
+        "model.layers.62.input_layernorm",
+        "model.layers.45.input_layernorm",
+        "model.layers.68.post_attention_layernorm",
+        "model.layers.72.input_layernorm",
+        "model.layers.32.mlp.gate.e_score_correction_bias",
+        "model.layers.27.input_layernorm",
+        "model.layers.16.post_attention_layernorm",
+        "model.layers.35.self_attn.q_a_layernorm",
+        "model.layers.58.self_attn.indexers_proj",
+        "model.layers.3.mlp.gate.e_score_correction_bias",
+        "model.layers.49.self_attn.q_a_layernorm",
+        "model.layers.66.mlp.gate.e_score_correction_bias",
+        "model.layers.10.self_attn.q_a_layernorm",
+        "model.layers.51.mlp.gate.e_score_correction_bias",
+        "model.layers.64.self_attn.q_a_layernorm",
+        "model.layers.70.input_layernorm",
+        "model.layers.71.post_attention_layernorm",
+        "model.layers.67.post_attention_layernorm",
+        "model.layers.6.self_attn.indexer.k_norm",
+        "model.layers.3.mlp.gate",
+        "model.layers.11.mlp.gate",
+        "model.layers.17.input_layernorm",
+        "model.layers.13.mlp.gate",
+        "model.layers.15.mlp.gate",
+        "model.layers.71.self_attn.q_a_layernorm",
+        "model.layers.33.mlp.gate.e_score_correction_bias",
+        "model.layers.72.mlp.gate.e_score_correction_bias",
+        "model.layers.65.post_attention_layernorm",
+        "model.layers.78.hnorm",
+        "model.layers.40.post_attention_layernorm",
+        "model.layers.12.mlp.gate.e_score_correction_bias",
+        "model.layers.8.input_layernorm",
+        "model.layers.73.mlp.gate",
+        "model.layers.11.post_attention_layernorm",
+        "model.layers.61.input_layernorm",
+        "model.layers.14.self_attn.kv_a_layernorm",
+        "model.layers.39.self_attn.kv_a_layernorm",
+        "model.layers.70.self_attn.indexer.k_norm",
+        "model.layers.45.mlp.gate",
+        "model.layers.51.self_attn.q_a_layernorm",
+        "model.layers.64.post_attention_layernorm",
+        "model.layers.20.post_attention_layernorm",
+        "model.layers.66.self_attn.kv_a_layernorm",
+        "model.layers.10.self_attn.indexer.k_norm.bias",
+        "model.layers.23.input_layernorm",
+        "model.layers.38.self_attn.indexer.k_norm",
+        "model.layers.19.self_attn.kv_a_layernorm",
+        "model.layers.40.self_attn.q_a_layernorm",
+        "model.layers.10.self_attn.indexer.k_norm",
+        "model.layers.44.post_attention_layernorm",
+        "model.layers.70.post_attention_layernorm",
+        "model.layers.41.mlp.gate.e_score_correction_bias",
+        "model.layers.36.post_attention_layernorm",
+        "model.layers.30.self_attn.indexer.k_norm.bias",
+        "model.layers.54.self_attn.q_a_layernorm",
+        "model.layers.59.mlp.gate",
+        "model.layers.69.post_attention_layernorm",
+        "lm_head",
+        "model.layers.77.self_attn.kv_a_layernorm",
+        "model.layers.75.self_attn.q_a_layernorm",
+        "model.layers.56.self_attn.q_a_layernorm",
+        "model.layers.50.self_attn.indexer.k_norm.bias",
+        "model.layers.54.mlp.gate",
+        "model.layers.6.self_attn.indexer.k_norm.bias",
+        "model.layers.15.input_layernorm",
+        "model.layers.29.self_attn.q_a_layernorm",
+        "model.layers.54.self_attn.indexers_proj",
+        "model.layers.73.self_attn.kv_a_layernorm",
+        "model.layers.0.self_attn.indexer.k_norm",
+        "model.layers.58.self_attn.indexer.k_norm",
+        "model.layers.36.input_layernorm",
+        "model.layers.47.self_attn.q_a_layernorm",
+        "model.layers.26.mlp.gate.e_score_correction_bias",
+        "model.layers.50.input_layernorm",
+        "model.layers.16.self_attn.q_a_layernorm",
+        "model.layers.19.mlp.gate.e_score_correction_bias",
+        "model.layers.78.input_layernorm",
+        "model.layers.27.mlp.gate",
+        "model.layers.19.self_attn.q_a_layernorm",
+        "model.layers.25.input_layernorm",
+        "model.layers.33.self_attn.kv_a_layernorm",
+        "model.layers.32.mlp.gate",
+        "model.layers.61.self_attn.kv_a_layernorm",
+        "model.layers.50.self_attn.indexers_proj",
+        "model.layers.34.self_attn.indexers_proj",
+        "model.layers.4.mlp.gate.e_score_correction_bias",
+        "model.layers.76.post_attention_layernorm",
+        "model.layers.57.post_attention_layernorm",
+        "model.layers.73.post_attention_layernorm",
+        "model.layers.68.mlp.gate.e_score_correction_bias",
+        "model.layers.34.self_attn.kv_a_layernorm",
+        "model.layers.78.enorm",
+        "model.layers.30.mlp.gate",
+        "model.layers.10.self_attn.indexers_proj",
+        "model.layers.34.input_layernorm",
+        "model.layers.34.post_attention_layernorm",
+        "model.layers.20.mlp.gate.e_score_correction_bias",
+        "model.layers.54.self_attn.indexer.k_norm.bias",
+        "model.layers.44.mlp.gate",
+        "model.layers.25.mlp.gate.e_score_correction_bias",
+        "model.layers.26.self_attn.indexers_proj",
+        "model.layers.66.self_attn.indexer.k_norm.bias",
+        "model.layers.41.post_attention_layernorm",
+        "model.layers.51.post_attention_layernorm",
+        "model.layers.22.self_attn.indexer.k_norm.bias",
+        "model.layers.21.mlp.gate.e_score_correction_bias",
+        "model.layers.15.self_attn.kv_a_layernorm",
+        "model.layers.25.self_attn.kv_a_layernorm",
+        "model.layers.55.mlp.gate",
+        "model.layers.11.self_attn.q_a_layernorm",
+        "model.layers.72.mlp.gate",
+        "model.layers.21.self_attn.kv_a_layernorm",
+        "model.layers.59.mlp.gate.e_score_correction_bias",
+        "model.layers.2.input_layernorm",
+        "model.layers.5.mlp.gate",
+        "model.layers.54.self_attn.kv_a_layernorm",
+        "model.layers.0.self_attn.indexers_proj",
+        "model.layers.71.mlp.gate.e_score_correction_bias",
+        "model.layers.6.self_attn.q_a_layernorm",
+        "model.layers.14.mlp.gate",
+        "model.layers.44.self_attn.kv_a_layernorm",
+        "model.layers.61.post_attention_layernorm",
+        "model.layers.64.self_attn.kv_a_layernorm",
+        "model.layers.1.post_attention_layernorm",
+        "model.layers.4.mlp.gate",
+        "model.layers.22.post_attention_layernorm",
+        "model.layers.21.self_attn.q_a_layernorm",
+        "model.layers.74.self_attn.q_a_layernorm",
+        "model.layers.18.self_attn.indexers_proj",
+        "model.layers.75.input_layernorm",
+        "model.layers.70.self_attn.q_a_layernorm",
+        "model.layers.22.self_attn.indexer.k_norm",
+        "model.layers.42.mlp.gate.e_score_correction_bias",
+        "model.layers.5.self_attn.kv_a_layernorm",
+        "model.layers.31.mlp.gate.e_score_correction_bias",
+        "model.layers.62.self_attn.indexer.k_norm.bias",
+        "model.layers.29.post_attention_layernorm",
+        "model.layers.6.self_attn.kv_a_layernorm",
+        "model.layers.48.input_layernorm",
+        "model.layers.74.self_attn.indexer.k_norm.bias",
+        "model.layers.7.input_layernorm",
+        "model.layers.39.self_attn.q_a_layernorm",
+        "model.layers.41.input_layernorm",
+        "model.layers.54.self_attn.indexer.k_norm",
+        "model.layers.32.self_attn.q_a_layernorm",
+        "model.layers.39.mlp.gate.e_score_correction_bias",
+        "model.layers.42.input_layernorm",
+        "model.layers.43.mlp.gate",
+        "model.layers.72.self_attn.q_a_layernorm",
+        "model.layers.37.mlp.gate.e_score_correction_bias",
+        "model.layers.5.post_attention_layernorm",
+        "model.layers.28.mlp.gate",
+        "model.layers.2.post_attention_layernorm",
+        "model.layers.49.post_attention_layernorm",
+        "model.layers.43.self_attn.kv_a_layernorm",
+        "model.layers.12.input_layernorm",
+        "model.layers.56.mlp.gate",
+        "model.layers.5.input_layernorm",
+        "model.layers.29.mlp.gate.e_score_correction_bias",
+        "model.layers.4.self_attn.kv_a_layernorm",
+        "model.layers.20.self_attn.q_a_layernorm",
+        "model.layers.59.self_attn.q_a_layernorm",
+        "model.layers.68.mlp.gate",
+        "model.layers.21.post_attention_layernorm",
+        "model.layers.26.self_attn.q_a_layernorm",
+        "model.layers.52.self_attn.q_a_layernorm",
+        "model.layers.29.self_attn.kv_a_layernorm",
+        "model.layers.24.input_layernorm",
+        "model.layers.34.self_attn.indexer.k_norm.bias",
+        "model.layers.43.input_layernorm",
+        "model.layers.27.self_attn.kv_a_layernorm",
+        "model.layers.31.input_layernorm",
+        "model.layers.69.mlp.gate",
+        "model.layers.26.post_attention_layernorm",
+        "model.layers.3.self_attn.q_a_layernorm",
+        "model.layers.37.post_attention_layernorm",
+        "model.layers.52.mlp.gate",
+        "model.layers.73.input_layernorm",
+        "model.layers.19.input_layernorm",
+        "model.layers.13.self_attn.q_a_layernorm",
+        "model.layers.76.input_layernorm",
+        "model.layers.8.mlp.gate",
+        "model.layers.63.post_attention_layernorm",
+        "model.layers.58.mlp.gate",
+        "model.layers.31.post_attention_layernorm",
+        "model.layers.23.mlp.gate.e_score_correction_bias",
+        "model.layers.34.self_attn.q_a_layernorm",
+        "model.layers.78.self_attn.indexer.k_norm.bias",
+        "model.layers.13.mlp.gate.e_score_correction_bias",
+        "model.layers.71.mlp.gate",
+        "model.layers.14.input_layernorm",
+        "model.layers.17.mlp.gate.e_score_correction_bias",
+        "model.layers.8.post_attention_layernorm",
+        "model.layers.45.self_attn.kv_a_layernorm",
+        "model.layers.46.self_attn.indexer.k_norm",
+        "model.layers.20.self_attn.kv_a_layernorm",
+        "model.layers.78.self_attn.indexer.k_norm",
+        "model.layers.37.mlp.gate",
+        "model.layers.7.mlp.gate",
+        "model.layers.30.mlp.gate.e_score_correction_bias",
+        "model.layers.67.mlp.gate",
+        "model.layers.73.self_attn.q_a_layernorm",
+        "model.layers.10.self_attn.kv_a_layernorm",
+        "model.layers.61.mlp.gate",
+        "model.layers.22.mlp.gate.e_score_correction_bias",
+        "model.layers.58.input_layernorm",
+        "model.layers.8.self_attn.kv_a_layernorm",
+        "model.layers.33.self_attn.q_a_layernorm",
+        "model.layers.55.self_attn.q_a_layernorm",
+        "model.layers.32.input_layernorm",
+        "model.layers.17.self_attn.kv_a_layernorm",
+        "model.layers.46.self_attn.indexer.k_norm.bias",
+        "model.layers.11.mlp.gate.e_score_correction_bias",
+        "model.layers.9.input_layernorm",
+        "model.layers.77.input_layernorm",
+        "model.layers.16.self_attn.kv_a_layernorm",
+        "model.layers.45.post_attention_layernorm",
+        "model.layers.74.mlp.gate",
+        "model.layers.68.self_attn.q_a_layernorm",
+        "model.layers.55.self_attn.kv_a_layernorm",
+        "model.layers.44.input_layernorm",
+        "model.layers.60.self_attn.kv_a_layernorm",
+        "model.layers.14.post_attention_layernorm",
+        "model.layers.62.self_attn.indexers_proj",
+        "model.layers.0.self_attn.q_a_layernorm",
+        "model.layers.72.post_attention_layernorm",
+        "model.layers.78.post_attention_layernorm",
+        "model.layers.60.mlp.gate.e_score_correction_bias",
+        "model.layers.26.self_attn.indexer.k_norm.bias",
+        "model.layers.38.post_attention_layernorm",
+        "model.layers.50.post_attention_layernorm",
+        "model.layers.28.input_layernorm",
+        "model.layers.30.self_attn.indexer.k_norm",
+        "model.layers.37.self_attn.q_a_layernorm",
+        "model.layers.56.input_layernorm",
+        "model.layers.27.post_attention_layernorm",
+        "model.layers.74.input_layernorm",
+        "model.layers.78.self_attn.q_a_layernorm",
+        "model.layers.42.self_attn.q_a_layernorm",
+        "model.layers.26.self_attn.indexer.k_norm",
+        "model.layers.42.self_attn.kv_a_layernorm",
+        "model.layers.1.self_attn.indexer.k_norm",
+        "model.layers.39.post_attention_layernorm",
+        "model.layers.48.mlp.gate.e_score_correction_bias",
+        "model.layers.74.post_attention_layernorm",
+        "model.layers.69.self_attn.q_a_layernorm",
+        "model.layers.13.input_layernorm",
+        "model.layers.74.self_attn.indexer.k_norm",
+        "model.layers.29.mlp.gate",
+        "model.layers.7.mlp.gate.e_score_correction_bias",
+        "model.layers.3.self_attn.kv_a_layernorm",
+        "model.layers.35.self_attn.kv_a_layernorm",
+        "model.layers.46.self_attn.indexers_proj",
+        "model.layers.0.self_attn.kv_a_layernorm",
+        "model.layers.43.self_attn.q_a_layernorm",
+        "model.norm",
+        "model.layers.1.self_attn.kv_a_layernorm",
+        "model.layers.38.self_attn.indexer.k_norm.bias",
+        "model.layers.30.self_attn.q_a_layernorm",
+        "model.layers.0.input_layernorm",
+        "model.layers.47.input_layernorm",
+        "model.layers.8.self_attn.q_a_layernorm",
+        "model.layers.61.self_attn.q_a_layernorm",
+        "model.layers.1.self_attn.indexer.k_norm.bias",
+        "model.layers.62.mlp.gate.e_score_correction_bias",
+        "model.layers.66.self_attn.q_a_layernorm",
+        "model.layers.46.self_attn.kv_a_layernorm",
+        "model.layers.66.mlp.gate",
+        "model.layers.69.input_layernorm",
+        "model.layers.69.self_attn.kv_a_layernorm",
+        "model.layers.22.input_layernorm",
+        "model.layers.30.post_attention_layernorm",
+        "model.layers.24.self_attn.q_a_layernorm",
+        "model.layers.48.mlp.gate",
+        "model.layers.7.self_attn.kv_a_layernorm",
+        "model.layers.67.input_layernorm",
+        "model.layers.27.self_attn.q_a_layernorm",
+        "model.layers.23.mlp.gate",
+        "model.layers.45.mlp.gate.e_score_correction_bias",
+        "model.layers.76.self_attn.kv_a_layernorm",
+        "model.layers.22.self_attn.q_a_layernorm",
+        "model.layers.78.mlp.gate",
+        "model.layers.9.mlp.gate",
+        "model.layers.57.self_attn.q_a_layernorm",
+        "model.layers.78.mlp.gate.e_score_correction_bias",
+        "model.layers.23.post_attention_layernorm",
+        "model.layers.38.self_attn.q_a_layernorm",
+        "model.layers.53.mlp.gate",
+        "model.layers.77.self_attn.q_a_layernorm",
+        "model.layers.62.self_attn.kv_a_layernorm",
+        "model.layers.12.mlp.gate",
+        "model.layers.34.self_attn.indexer.k_norm",
+        "model.layers.46.self_attn.q_a_layernorm",
+        "model.layers.36.self_attn.q_a_layernorm",
+        "model.layers.58.self_attn.indexer.k_norm.bias",
+        "model.layers.60.post_attention_layernorm",
+        "model.layers.28.post_attention_layernorm",
+        "model.layers.2.self_attn.kv_a_layernorm",
+        "model.layers.70.self_attn.indexers_proj",
+        "model.layers.16.mlp.gate.e_score_correction_bias",
+        "model.layers.2.self_attn.indexer.k_norm",
+        "model.layers.31.self_attn.kv_a_layernorm",
+        "model.layers.64.input_layernorm",
+        "model.layers.6.mlp.gate.e_score_correction_bias",
+        "model.layers.50.self_attn.indexer.k_norm",
+        "model.layers.58.self_attn.q_a_layernorm",
+        "model.layers.57.mlp.gate.e_score_correction_bias",
+        "model.layers.13.post_attention_layernorm",
+        "model.layers.28.self_attn.kv_a_layernorm",
+        "model.layers.31.self_attn.q_a_layernorm",
+        "model.layers.1.self_attn.q_a_layernorm",
+        "model.layers.17.mlp.gate",
+        "model.layers.15.post_attention_layernorm",
+        "model.layers.18.self_attn.indexer.k_norm.bias",
+        "model.layers.18.self_attn.indexer.k_norm",
+        "model.layers.43.mlp.gate.e_score_correction_bias",
+        "model.layers.11.input_layernorm",
+        "model.layers.9.post_attention_layernorm",
+        "model.layers.33.input_layernorm",
+        "model.layers.66.input_layernorm",
+        "model.layers.78.self_attn.indexers_proj",
+        "model.layers.54.post_attention_layernorm",
+        "model.layers.53.mlp.gate.e_score_correction_bias",
+        "model.layers.5.mlp.gate.e_score_correction_bias",
+        "model.layers.19.mlp.gate",
+        "model.layers.49.mlp.gate",
+        "model.layers.66.self_attn.indexers_proj",
+        "model.layers.14.self_attn.indexer.k_norm.bias",
+        "model.layers.41.mlp.gate",
+        "model.layers.42.self_attn.indexer.k_norm.bias",
+        "model.layers.65.input_layernorm",
+        "model.layers.75.mlp.gate",
+        "model.layers.12.self_attn.kv_a_layernorm",
+        "model.layers.53.input_layernorm",
+        "model.layers.58.self_attn.kv_a_layernorm",
+        "model.layers.60.self_attn.q_a_layernorm",
+        "model.layers.6.mlp.gate",
+        "model.layers.49.self_attn.kv_a_layernorm",
+        "model.layers.16.mlp.gate",
+        "model.layers.22.mlp.gate",
+        "model.layers.58.post_attention_layernorm",
+        "model.layers.15.mlp.gate.e_score_correction_bias",
+        "model.layers.26.input_layernorm",
+        "model.layers.3.post_attention_layernorm",
+        "model.layers.6.input_layernorm",
+        "model.layers.22.self_attn.kv_a_layernorm",
+        "model.layers.16.input_layernorm",
+        "model.layers.33.post_attention_layernorm",
+        "model.layers.17.post_attention_layernorm",
+        "model.layers.35.mlp.gate",
+        "model.layers.56.post_attention_layernorm",
+        "model.layers.7.self_attn.q_a_layernorm",
+        "model.layers.24.mlp.gate",
+        "model.layers.24.post_attention_layernorm",
+        "model.layers.36.mlp.gate.e_score_correction_bias",
+        "model.layers.56.mlp.gate.e_score_correction_bias",
+        "model.layers.32.post_attention_layernorm",
+        "model.layers.47.post_attention_layernorm",
+        "model.layers.40.mlp.gate",
+        "model.layers.41.self_attn.kv_a_layernorm",
+        "model.layers.52.self_attn.kv_a_layernorm",
+        "model.layers.9.self_attn.kv_a_layernorm",
+        "model.layers.77.mlp.gate.e_score_correction_bias",
+        "model.layers.78.shared_head.norm",
+        "model.layers.53.post_attention_layernorm",
+        "model.layers.63.self_attn.kv_a_layernorm",
+        "model.layers.48.self_attn.kv_a_layernorm",
+        "model.layers.28.mlp.gate.e_score_correction_bias",
+        "model.layers.8.mlp.gate.e_score_correction_bias",
+        "model.layers.52.mlp.gate.e_score_correction_bias",
+        "model.layers.52.post_attention_layernorm",
+        "model.layers.18.self_attn.q_a_layernorm",
+        "model.layers.50.mlp.gate.e_score_correction_bias",
+        "model.layers.64.mlp.gate.e_score_correction_bias",
+        "model.layers.38.mlp.gate.e_score_correction_bias",
+        "model.layers.59.input_layernorm",
+        "model.layers.21.input_layernorm",
+        "model.layers.39.mlp.gate",
+        "model.layers.77.post_attention_layernorm",
+        "model.layers.76.self_attn.q_a_layernorm",
+        "model.layers.12.post_attention_layernorm",
+        "model.layers.43.post_attention_layernorm",
+        "model.layers.35.input_layernorm",
+        "model.layers.62.mlp.gate",
+        "model.layers.38.self_attn.kv_a_layernorm",
+        "model.layers.63.mlp.gate",
+        "model.layers.27.mlp.gate.e_score_correction_bias",
+        "model.layers.18.mlp.gate",
+        "model.layers.25.post_attention_layernorm",
+        "model.layers.55.mlp.gate.e_score_correction_bias",
+        "model.layers.66.self_attn.indexer.k_norm",
+        "model.layers.48.self_attn.q_a_layernorm",
+        "model.layers.78.eh_proj",
+        "model.layers.77.mlp.gate",
+        "model.layers.6.self_attn.indexers_proj",
+        "model.layers.14.self_attn.q_a_layernorm",
+        "model.layers.2.self_attn.q_a_layernorm",
+        "model.layers.42.mlp.gate",
+        "model.layers.54.mlp.gate.e_score_correction_bias",
+        "model.layers.60.input_layernorm",
+        "model.layers.40.self_attn.kv_a_layernorm",
+        "model.layers.51.self_attn.kv_a_layernorm",
+        "model.layers.18.input_layernorm",
+        "model.layers.30.self_attn.indexers_proj",
+        "model.layers.53.self_attn.kv_a_layernorm",
+        "model.layers.6.post_attention_layernorm",
+        "model.layers.70.self_attn.indexer.k_norm.bias",
+        "model.layers.54.input_layernorm",
+        "model.layers.41.self_attn.q_a_layernorm",
+        "model.layers.18.mlp.gate.e_score_correction_bias",
+        "model.layers.34.mlp.gate.e_score_correction_bias",
+        "model.layers.47.mlp.gate",
+        "model.layers.46.mlp.gate",
+        "model.layers.25.self_attn.q_a_layernorm",
+        "model.layers.63.mlp.gate.e_score_correction_bias",
+        "model.layers.66.post_attention_layernorm",
+        "model.layers.76.mlp.gate",
+        "model.embed_tokens",
+        "model.layers.67.self_attn.kv_a_layernorm",
+        "model.layers.67.mlp.gate.e_score_correction_bias",
+        "model.layers.26.self_attn.kv_a_layernorm",
+        "model.layers.31.mlp.gate",
+        "model.layers.18.self_attn.kv_a_layernorm",
+        "model.layers.34.mlp.gate",
+        "model.layers.55.post_attention_layernorm",
+        "model.layers.30.input_layernorm",
+        "model.layers.50.mlp.gate",
+        "model.layers.63.input_layernorm",
+        "model.layers.11.self_attn.kv_a_layernorm",
+        "model.layers.24.self_attn.kv_a_layernorm",
+        "model.layers.75.mlp.gate.e_score_correction_bias",
+        "model.layers.10.post_attention_layernorm",
+        "model.layers.1.self_attn.indexers_proj",
+        "model.layers.42.self_attn.indexers_proj",
+        "model.layers.46.input_layernorm",
+        "model.layers.23.self_attn.q_a_layernorm",
+        "model.layers.63.self_attn.q_a_layernorm",
+        "model.layers.14.self_attn.indexer.k_norm",
+        "model.layers.14.self_attn.indexers_proj",
+        "model.layers.60.mlp.gate",
+        "model.layers.70.mlp.gate.e_score_correction_bias",
+        "model.layers.53.self_attn.q_a_layernorm",
+        "model.layers.0.post_attention_layernorm",
+        "model.layers.35.mlp.gate.e_score_correction_bias",
+        "model.layers.75.self_attn.kv_a_layernorm",
+        "model.layers.1.input_layernorm",
+        "model.layers.42.post_attention_layernorm",
+        "model.layers.37.input_layernorm",
+        "model.layers.9.mlp.gate.e_score_correction_bias",
+        "model.layers.46.post_attention_layernorm",
+        "model.layers.26.mlp.gate",
+        "model.layers.65.mlp.gate.e_score_correction_bias",
+        "model.layers.67.self_attn.q_a_layernorm",
+        "model.layers.44.mlp.gate.e_score_correction_bias",
+        "model.layers.33.mlp.gate",
+        "model.layers.29.input_layernorm",
+        "model.layers.18.post_attention_layernorm",
+        "model.layers.75.post_attention_layernorm",
+        "model.layers.44.self_attn.q_a_layernorm",
+        "model.layers.10.mlp.gate",
+        "model.layers.21.mlp.gate",
+        "model.layers.58.mlp.gate.e_score_correction_bias",
+        "model.layers.7.post_attention_layernorm",
+        "model.layers.50.self_attn.q_a_layernorm",
+        "model.layers.9.self_attn.q_a_layernorm",
+        "model.layers.2.self_attn.indexers_proj",
+        "model.layers.0.self_attn.indexer.k_norm.bias",
+        "model.layers.74.self_attn.kv_a_layernorm",
+        "model.layers.74.mlp.gate.e_score_correction_bias",
+        "model.layers.49.mlp.gate.e_score_correction_bias",
+        "model.layers.62.post_attention_layernorm",
+        "model.layers.57.self_attn.kv_a_layernorm",
+        "model.layers.40.input_layernorm",
+        "model.layers.69.mlp.gate.e_score_correction_bias",
+        "model.layers.35.post_attention_layernorm",
+        "model.layers.74.self_attn.indexers_proj",
+        "model.layers.55.input_layernorm",
+        "model.layers.49.input_layernorm",
+        "model.layers.40.mlp.gate.e_score_correction_bias",
+        "model.layers.25.mlp.gate",
+        "model.layers.2.self_attn.indexer.k_norm.bias",
+        "model.layers.68.self_attn.kv_a_layernorm",
+        "model.layers.51.input_layernorm",
+        "model.layers.22.self_attn.indexers_proj",
+        "model.layers.10.input_layernorm",
+        "model.layers.42.self_attn.indexer.k_norm",
+        "model.layers.56.self_attn.kv_a_layernorm",
+        "model.layers.17.self_attn.q_a_layernorm",
+        "model.layers.59.self_attn.kv_a_layernorm",
+        "model.layers.3.input_layernorm",
+        "model.layers.57.input_layernorm",
+        "model.layers.38.input_layernorm",
+        "model.layers.62.self_attn.indexer.k_norm",
+        "model.layers.20.mlp.gate",
+        "model.layers.64.mlp.gate",
+        "model.layers.28.self_attn.q_a_layernorm",
+        "model.layers.32.self_attn.kv_a_layernorm",
+        "model.layers.10.mlp.gate.e_score_correction_bias",
+        "model.layers.5.self_attn.q_a_layernorm",
+        "model.layers.37.self_attn.kv_a_layernorm"
+      ]
+    },
+    "mtp_bits": 16,
+    "scope": "glm53_routed_experts_only",
+    "keys_profile": "routed-experts-3bpw-mul1; attn/dense/shared/head bf16",
+    "keys_source": "zai-org/GLM-5.3 (FP8) via exllamav3 1.4.5 + keys encode farm"
+  }
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "pad_token_id": 154820,
+  "temperature": 1.0,
+  "top_p": 0.95,
+  "transformers_version": "5.12.0"
+}

model-00001-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fec34deec06cd68b9ced94494ab653836c7e645bf98dadee46cdf1e98432ca2b
+size 8357250403

model-00005-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:966889d2a98cede970ea40b1d006ebcd40d24e372b81ed1cac6b807340b645b2
+size 8111403478

model-00009-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cdf80c8975f78f82ef7200be56f0ef342cc534b01b5d4c0ba94b9483a8ad46af
+size 8111403478

model-00010-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b383a7f2c463d59745dbcaf9883d656328550611f84475ce83e035b848c37db
+size 8092659062

model-00011-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6488dcf79d89daee948e4ea43c54e8eb4ec23857f12a1a874c04870f7a5700a2
+size 8111403478

model-00013-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed1ba921b27c719cf09cefa0f80a92deee6633964bd6990612a142e2378cb6f2
+size 8111403478

model-00014-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92ecb399799bb0d1d11c8b4469f80c4f84cf63d086ce5d38dc9e7bf65a97819c
+size 8092659062

model-00019-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c12183bb06e161c2df208eb208a80729d3034ab1c02607e5ebd44384709096c
+size 8111403478

model-00020-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe76908fb0495d600f2cca59d1d6802d3529d74959e9685712a9eba025817c40
+size 8092659062

model-00021-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8f4340829f726f92fdb5b10823ffe0f4a4578db1d73b8eeefdeeff45b3e3261
+size 8111403478

model-00023-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ba92d7e5210f9565ac587f776cbaed146febbf6b9ce1f2de9b835c14a6b6cf3
+size 8111403478

model-00024-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dca97644a9978c7f42b498bb84df2bc81af17f9b55000781a26d992bf2c2052c
+size 8092659062

model-00030-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f0c68eab43e468fa8ae8607a069ecb02bca8b670645322cbe56dacd3eb38916
+size 8092659062

model-00031-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ccc7fb59c8f4a3c81b3d4d64c6d0880aee12e1748c750f351aa8c80410dc581
+size 8111403478

model-00032-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98525b8f4757949a7e60ec075c5dec20b2d2208f220a673b3db8c89d31e1823d
+size 8092659062

model-00034-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:72777c7367f3973d9d10ca88e8fe95896d65342bc8a27511f057079f67599911
+size 8092659062

model-00035-of-00041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8f21f16c2a599412edecfaee695d48cb0c2ac7ffa6b18fc37941f30137aa7b3
+size 8111403478

quant/build.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+# exllamav3 1.4.5 native sm_121a build inside the Mia image (run via docker exec)
+set -x
+CU=/usr/local/lib/python3.12/dist-packages/nvidia/cu13/include
+for h in $CU/*.h; do b=$(basename $h); [ -e /usr/local/cuda/include/$b ] || ln -s $h /usr/local/cuda/include/$b; done
+cd /work/exllamav3
+unset EXLLAMA_NOCOMPILE
+export TORCH_CUDA_ARCH_LIST=12.1a MAX_JOBS=20
+pip install --no-deps --no-build-isolation -v . 2>&1 | tail -40
+echo BUILD_EXIT=${PIPESTATUS[0]}
+ls -la /usr/local/lib/python3.12/dist-packages/exllamav3_ext*.so
+python3 -c "import exllamav3, exllamav3.ext; print(\"IMPORT_OK\", exllamav3.__version__)"
+python3 -c "from exllamav3.architecture.glm_moe_dsa import GlmMoeDsaConfig; print(\"ARCH_OK\")"
+echo BUILD_DONE

quant/exl3_farm.py

@@ -0,0 +1,185 @@
+#!/usr/bin/env python3
+"""EXL3 encode farm: run exllamav3's trellis quantization of individual linears on remote GPUs.
+
+The converter's layer chain is sequential (each layer is calibrated on the quantized output of the
+previous ones), but inside a layer every expert matrix is encoded independently given its Hessian.
+This module lets the master (running convert.py) ship (weights, finalized H_data, quant_args) for a
+group of linears to a worker GPU on another node and get back the packed EXL3 tensors.
+
+Worker:  python3 exl3_farm.py serve --port 5599
+Client:  quantize_remote(host, port, weights, H_datas, quant_args_list) -> (results, quant_args_list)
+
+Wire format: 8-byte little-endian length + torch.save payload (pickled dict of CPU tensors).
+"""
+import argparse, io, socket, struct, sys, time, threading, traceback
+
+import torch
+
+MAGIC_OK = b"OK"
+MAGIC_ERR = b"ER"
+
+
+def _send(sock: socket.socket, obj) -> None:
+    buf = io.BytesIO()
+    torch.save(obj, buf)
+    data = buf.getvalue()
+    sock.sendall(struct.pack("<Q", len(data)))
+    sock.sendall(data)
+
+
+def _recv_exact(sock: socket.socket, n: int) -> bytes:
+    chunks = []
+    got = 0
+    while got < n:
+        c = sock.recv(min(n - got, 64 << 20))
+        if not c:
+            raise ConnectionError("socket closed")
+        chunks.append(c)
+        got += len(c)
+    return b"".join(chunks)
+
+
+def _recv(sock: socket.socket, map_location="cpu"):
+    n = struct.unpack("<Q", _recv_exact(sock, 8))[0]
+    data = _recv_exact(sock, n)
+    return torch.load(io.BytesIO(data), map_location=map_location, weights_only=False)
+
+
+# ------------------------------------------------------------------ client (master side)
+_H_KEYS = ("H", "first_key", "count", "finalized", "num_total", "inf_nan", "su", "L", "diag", "q_fallback")
+
+
+def _export_H(hd: dict) -> dict:
+    out = {}
+    for k in _H_KEYS:
+        v = hd.get(k)
+        if isinstance(v, torch.Tensor):
+            v = v.detach().to("cpu")
+        out[k] = v
+    return out
+
+
+def quantize_remote(host: str, port: int, weights, H_datas, quant_args_list, timeout=3600):
+    """Quantize a group of linears on a remote worker. weights: list of (in,out) fp16/fp32 tensors
+    (CPU ok); H_datas: list of FINALIZED H_data dicts (may repeat the same object); returns
+    ([(proxy_err, out_tensors_cpu), ...], updated quant_args_list)."""
+    pool, idx = [], []
+    for hd in H_datas:
+        for i, p in enumerate(pool):
+            if p is hd:
+                idx.append(i)
+                break
+        else:
+            pool.append(hd)
+            idx.append(len(pool) - 1)
+    req = {
+        "weights": [w.detach().to("cpu") for w in weights],
+        "H_pool": [_export_H(hd) for hd in pool],
+        "H_idx": idx,
+        "quant_args_list": [{k: v for k, v in qa.items() if k != "devices"} for qa in quant_args_list],
+    }
+    sock = socket.create_connection((host, port), timeout=timeout)
+    try:
+        sock.settimeout(timeout)
+        _send(sock, req)
+        rep = _recv(sock, map_location="cpu")
+    finally:
+        sock.close()
+    if rep.get("status") != "ok":
+        raise RuntimeError(f"remote {host}:{port} failed: {rep.get('error')}")
+    for qa, nqa in zip(quant_args_list, rep["quant_args_list"]):
+        for k, v in nqa.items():
+            if k != "devices":
+                qa[k] = v
+    return rep["results"], quant_args_list
+
+
+# ------------------------------------------------------------------ server (worker side)
+def _handle(conn: socket.socket, device: torch.device, log):
+    from exllamav3.modules.quant.exl3_lib.quantize import quantize_exl3, quantize_exl3_batch
+
+    t0 = time.time()
+    req = _recv(conn, map_location="cpu")
+    t1 = time.time()
+    try:
+        with torch.inference_mode():
+            pool = req["H_pool"]
+            for hd in pool:
+                for k, v in list(hd.items()):
+                    if isinstance(v, torch.Tensor):
+                        hd[k] = v.to(device)
+                hd["device"] = device
+            H_datas = [pool[i] for i in req["H_idx"]]
+            weights = req["weights"]
+            qal = req["quant_args_list"]
+            for qa in qal:
+                qa["devices"] = [device.index or 0]
+                qa["device_ratios"] = None
+                qa["debug_dir"] = "/tmp/exl3-farm-debug"
+            if len(weights) > 1:
+                results = quantize_exl3_batch(weights, H_datas, qal)
+            else:
+                _, pe, out = quantize_exl3(weights[0].to(device).float(), H_datas[0], qal[0], False)
+                results = [(pe, out)]
+            cpu_results = []
+            for pe, out in results:
+                cpu_results.append((float(pe), {k: v.detach().to("cpu") for k, v in out.items() if v is not None}))
+            rep = {
+                "status": "ok",
+                "results": cpu_results,
+                "quant_args_list": [{k: (v.detach().to("cpu") if isinstance(v, torch.Tensor) else v)
+                                     for k, v in qa.items() if k not in ("devices",)} for qa in qal],
+            }
+        torch.cuda.synchronize(device)
+    except Exception as e:  # noqa: BLE001
+        rep = {"status": "error", "error": f"{e!r}\n{traceback.format_exc()}"}
+        log(f"ERROR: {e!r}")
+    t2 = time.time()
+    _send(conn, rep)
+    n = len(req["weights"])
+    log(f"group of {n}: recv {t1 - t0:.2f}s quant {t2 - t1:.2f}s send {time.time() - t2:.2f}s")
+
+
+def serve(port: int, device_index: int, concurrency: int):
+    device = torch.device(f"cuda:{device_index}")
+    torch.cuda.set_device(device)
+    lock = threading.Semaphore(concurrency)
+
+    def log(msg):
+        print(f"[farm {time.strftime('%H:%M:%S')}] {msg}", flush=True)
+
+    # warm the kernels once so the first real request does not pay JIT/allocation
+    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+    srv.bind(("0.0.0.0", port))
+    srv.listen(8)
+    log(f"listening on :{port} device={device} concurrency={concurrency}")
+
+    def worker(conn):
+        with lock:
+            try:
+                _handle(conn, device, log)
+            except Exception as e:  # noqa: BLE001
+                log(f"connection failed: {e!r}")
+            finally:
+                conn.close()
+
+    while True:
+        conn, addr = srv.accept()
+        conn.settimeout(3600)
+        threading.Thread(target=worker, args=(conn,), daemon=True).start()
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser()
+    sub = ap.add_subparsers(dest="cmd")
+    s = sub.add_parser("serve")
+    s.add_argument("--port", type=int, default=5599)
+    s.add_argument("--device", type=int, default=0)
+    s.add_argument("--concurrency", type=int, default=1)
+    a = ap.parse_args()
+    if a.cmd == "serve":
+        serve(a.port, a.device, a.concurrency)
+    else:
+        ap.print_help()
+        sys.exit(1)

quant/make_recipe.py

@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+"""Generate the exllamav3 per-tensor recipe for a Mia-native GLM-5.3 EXL3 profile.
+
+Profile (matches MiaAI-Lab GLM-5.3-Flash EXL3 scope=glm53_routed_experts_only):
+  * routed experts gate/up/down (layers first_k_dense..77) -> EXPERT_BITS (uniform, mcg codebook)
+  * everything else that exllamav3 budgets (MLA attention projections, indexer wq_b,
+    dense MLP layers 0-2, shared experts) -> 16 (stored unquantized)
+  * lm_head -> --head_bits 16, MTP layer 78 -> --mtp_bits 16 (not in the recipe)
+Usage: make_recipe.py <model.safetensors.index.json> <config.json> <out.yaml> [expert_bits]
+"""
+import json, re, sys
+
+index, config, out = sys.argv[1:4]
+EXPERT_BITS = int(sys.argv[4]) if len(sys.argv) > 4 else 3
+
+cfg = json.load(open(config))
+n_layers = cfg["num_hidden_layers"]
+first_dense = cfg.get("first_k_dense_replace", 3)
+wm = json.load(open(index))["weight_map"]
+
+NONEXPERT = {
+    "self_attn.q_a_proj", "self_attn.q_b_proj", "self_attn.kv_a_proj_with_mqa",
+    "self_attn.o_proj", "self_attn.indexer.wq_b",
+    "mlp.gate_proj", "mlp.up_proj", "mlp.down_proj",
+    "mlp.shared_experts.gate_proj", "mlp.shared_experts.up_proj", "mlp.shared_experts.down_proj",
+}
+EXPERT_RE = re.compile(r"mlp\.experts\.(\d+)\.(gate|up|down)_proj$")
+
+tensors = {}
+for k in wm:
+    if not k.endswith(".weight"):
+        continue
+    m = re.match(r"model\.layers\.(\d+)\.(.*)\.weight$", k)
+    if not m:
+        continue
+    layer, rest = int(m.group(1)), m.group(2)
+    if layer >= n_layers:          # MTP layer(s): governed by --mtp_bits
+        continue
+    key = f"model.layers.{layer}.{rest}"
+    if EXPERT_RE.match(rest):
+        assert layer >= first_dense, key
+        tensors[key] = EXPERT_BITS
+    elif rest in NONEXPERT:
+        tensors[key] = 16
+
+n_exp = sum(1 for v in tensors.values() if v == EXPERT_BITS)
+n_16 = sum(1 for v in tensors.values() if v == 16)
+with open(out, "w") as f:
+    f.write(f"# GLM-5.3 (753B) Mia-native EXL3 recipe: routed experts {EXPERT_BITS}b mcg, rest bf16\n")
+    f.write(f"target_bpw: {EXPERT_BITS}\nhead_bits: 16\ntensors:\n")
+    for k in sorted(tensors, key=lambda s: (int(s.split('.')[2]), s)):
+        f.write(f'  "{k}": {tensors[k]}\n')
+print(f"recipe: {len(tensors)} tensors  experts@{EXPERT_BITS}b={n_exp}  unquantized(16)={n_16}  -> {out}")

quant/orchestrate.py

@@ -0,0 +1,378 @@
+#!/usr/bin/env python3
+"""Rolling-shard driver for exllamav3 convert.py on a disk-limited node (stdlib only, runs on host).
+
+Why: the FP8 source (zai-org/GLM-5.3, 141 shards, 756 GB) does not fit next to the ~300 GB work
+dir and ~300 GB output. exllamav3 walks modules in order and checkpoints per module, so we:
+  1. keep a rolling window of source shards (download ahead in a thread),
+  2. run convert.py in a fresh container per step with --max_module so it stops after layer L,
+  3. between steps, rewrite finished shards dropping only `model.layers.{L}.mlp.experts.*`
+     (norms/attn/shared/router stay, compile needs them), then advance.
+Every convert process rescans the source dir at start, so rewritten shards are always consistent.
+
+Layout (ROOT): src/ (rolling source), work/ (exllamav3 work dir), out/ (final), runs/ (logs),
+state.json (progress), shards.json (HF sizes + sha256).
+"""
+import json, os, re, sys, time, struct, subprocess, threading, hashlib, shutil, urllib.request
+
+ROOT = os.environ.get("ROOT", "/home/keyspark/glm53-full-exl3")
+SRC, WORK, OUT, RUNS = (os.path.join(ROOT, d) for d in ("src", "work", "out", "runs"))
+STUB = os.path.join(ROOT, "src-stub")
+STATE = os.path.join(ROOT, "state.json")
+LOG = os.path.join(ROOT, "orchestrate.log")
+REPO = "zai-org/GLM-5.3"
+HF = f"https://huggingface.co/{REPO}/resolve/main/"
+IMAGE = os.environ.get("IMAGE", "glm53-exl3-quant:1.4.5")
+CONTAINER = "glm53-exl3-convert"
+RECIPE = os.path.join(ROOT, "recipe.yaml")
+EXPERT_BITS = int(os.environ.get("EXPERT_BITS", "3"))
+LAYERS_PER_RUN = int(os.environ.get("LAYERS_PER_RUN", "2"))
+WINDOW = int(os.environ.get("WINDOW", "5"))          # layers downloaded ahead of the current one
+MIN_FREE_GB = int(os.environ.get("MIN_FREE_GB", "80"))
+MEM_CAP = os.environ.get("MEM_CAP", "112g")
+NUM_LAYERS = 78
+MTP_LAYER = 78
+
+def log(msg):
+    line = f"[{time.strftime('%m-%d %H:%M:%S')}] {msg}"
+    print(line, flush=True)
+    with open(LOG, "a") as f:
+        f.write(line + "\n")
+
+def free_gb(path=ROOT):
+    st = os.statvfs(path)
+    return st.f_bavail * st.f_frsize / 1e9
+
+# ---------------------------------------------------------------- index / shard maps
+index = json.load(open(os.path.join(STUB, "model.safetensors.index.json")))["weight_map"]
+layer_shards = {}          # layer -> set(shard)
+shard_layers = {}          # shard -> set(layer)
+special = set()            # shards holding embed/lm_head/norm
+for k, s in index.items():
+    m = re.match(r"model\.layers\.(\d+)\.", k)
+    if m:
+        L = int(m.group(1))
+        layer_shards.setdefault(L, set()).add(s)
+        shard_layers.setdefault(s, set()).add(L)
+    else:
+        special.add(s)
+        shard_layers.setdefault(s, set())
+ALL_SHARDS = sorted(shard_layers)
+PERMANENT = set(special) | layer_shards[MTP_LAYER]   # eh_proj must exist at every resume
+
+def load_state():
+    if os.path.exists(STATE):
+        return json.load(open(STATE))
+    return {"done_layers": [], "dropped": {}, "started": False, "finished": False}
+
+def save_state(st):
+    tmp = STATE + ".tmp"
+    json.dump(st, open(tmp, "w"), indent=1)
+    os.replace(tmp, STATE)
+
+# ---------------------------------------------------------------- HF shard metadata
+def shard_meta():
+    p = os.path.join(ROOT, "shards.json")
+    if os.path.exists(p):
+        return json.load(open(p))
+    with urllib.request.urlopen(f"https://huggingface.co/api/models/{REPO}?blobs=true", timeout=60) as r:
+        d = json.load(r)
+    meta = {}
+    for s in d["siblings"]:
+        fn = s["rfilename"]
+        if fn.endswith(".safetensors"):
+            lfs = s.get("lfs") or {}
+            meta[fn] = {"size": s.get("size") or lfs.get("size"), "sha256": lfs.get("oid")}
+    json.dump(meta, open(p, "w"), indent=1)
+    return meta
+
+META = shard_meta()
+
+# ---------------------------------------------------------------- download (thread)
+dl_lock = threading.Lock()
+dl_inflight = set()
+
+def sha256_file(path):
+    h = hashlib.sha256()
+    with open(path, "rb") as f:
+        for chunk in iter(lambda: f.read(64 << 20), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+def download(shard):
+    dst = os.path.join(SRC, shard)
+    part = dst + ".part"
+    if os.path.exists(dst):
+        return True
+    # one curl per shard: the prefetch thread and the main thread may want the same file
+    with dl_lock:
+        if shard in dl_inflight:
+            busy = True
+        else:
+            dl_inflight.add(shard)
+            busy = False
+    if busy:
+        while shard in dl_inflight:
+            time.sleep(5)
+        return os.path.exists(dst)
+    try:
+        return _download(shard, dst, part)
+    finally:
+        with dl_lock:
+            dl_inflight.discard(shard)
+
+def _download(shard, dst, part):
+    for attempt in range(5):
+        rc = subprocess.call(["curl", "-sSL", "-C", "-", "--retry", "8", "--retry-delay", "5",
+                              "-o", part, HF + shard])
+        if rc == 0 and os.path.exists(part):
+            size = os.path.getsize(part)
+            want = META.get(shard, {}).get("size")
+            if want and size != want:
+                log(f"  size mismatch {shard}: {size} != {want}, retrying")
+                os.remove(part)
+                continue
+            want_sha = META.get(shard, {}).get("sha256")
+            if want_sha:
+                got = sha256_file(part)
+                if got != want_sha:
+                    log(f"  sha256 mismatch {shard}, retrying")
+                    os.remove(part)
+                    continue
+            os.replace(part, dst)
+            log(f"  downloaded {shard} ({size/1e9:.1f} GB) free={free_gb():.0f} GB")
+            return True
+        time.sleep(10)
+    log(f"!! download failed for {shard}")
+    return False
+
+def needed_shards(upto_layer):
+    """Shards required for layers <= upto_layer that are not done, plus permanent ones."""
+    st = load_state()
+    done = set(st["done_layers"])
+    need = set(PERMANENT)
+    for L in range(0, min(upto_layer, NUM_LAYERS - 1) + 1):
+        if L not in done:
+            need |= layer_shards[L]
+    return need
+
+def prefetch_loop(get_current):
+    while True:
+        try:
+            st = load_state()
+            if st.get("finished"):
+                return
+            cur = get_current()
+            for shard in sorted(needed_shards(cur + WINDOW)):
+                dst = os.path.join(SRC, shard)
+                if os.path.exists(dst):
+                    continue
+                if free_gb() < MIN_FREE_GB:
+                    break
+                download(shard)
+        except Exception as e:
+            log(f"prefetch error: {e!r}")
+        time.sleep(15)
+
+# ---------------------------------------------------------------- shard rewrite (pure python)
+def rewrite_shard(path, drop_prefixes):
+    """Rewrite a safetensors file keeping only tensors not starting with drop_prefixes."""
+    with open(path, "rb") as f:
+        n = struct.unpack("<Q", f.read(8))[0]
+        hdr = json.loads(f.read(n))
+        base = 8 + n
+        meta = hdr.pop("__metadata__", None)
+        keep = {k: v for k, v in hdr.items() if not any(k.startswith(p) for p in drop_prefixes)}
+        if len(keep) == len(hdr):
+            return False
+        items = sorted(keep.items(), key=lambda kv: kv[1]["data_offsets"][0])
+        new = {}
+        off = 0
+        for k, v in items:
+            s, e = v["data_offsets"]
+            new[k] = {"dtype": v["dtype"], "shape": v["shape"], "data_offsets": [off, off + (e - s)]}
+            off += e - s
+        if meta is not None:
+            new["__metadata__"] = meta
+        hb = json.dumps(new, separators=(",", ":")).encode()
+        hb += b" " * ((8 - len(hb) % 8) % 8)
+        tmp = path + ".rewrite"
+        with open(tmp, "wb") as o:
+            o.write(struct.pack("<Q", len(hb)))
+            o.write(hb)
+            for k, v in items:
+                s, e = v["data_offsets"]
+                f.seek(base + s)
+                remaining = e - s
+                while remaining:
+                    chunk = f.read(min(64 << 20, remaining))
+                    o.write(chunk)
+                    remaining -= len(chunk)
+    before = os.path.getsize(path)
+    os.replace(tmp, path)
+    log(f"  rewrote {os.path.basename(path)}: {before/1e9:.1f} -> {os.path.getsize(path)/1e9:.1f} GB")
+    return True
+
+def drop_done_experts(st):
+    done = set(st["done_layers"])
+    for shard in ALL_SHARDS:
+        path = os.path.join(SRC, shard)
+        if not os.path.exists(path):
+            continue
+        already = set(st["dropped"].get(shard, []))
+        todo = sorted((shard_layers[shard] & done) - already)
+        todo = [L for L in todo if L != MTP_LAYER or st.get("mtp_done")]
+        if not todo:
+            continue
+        prefixes = [f"model.layers.{L}.mlp.experts." for L in todo]
+        rewrite_shard(path, prefixes)
+        st["dropped"][shard] = sorted(already | set(todo))
+        save_state(st)
+
+# ---------------------------------------------------------------- convert runs
+def image_ready():
+    return subprocess.call(["docker", "image", "inspect", IMAGE], stdout=subprocess.DEVNULL,
+                           stderr=subprocess.DEVNULL) == 0
+
+def run_convert(max_module, first, tag):
+    os.makedirs(RUNS, exist_ok=True)
+    logf = os.path.join(RUNS, f"run-{tag}.log")
+    cmd = ["docker", "run", "--rm", "--name", CONTAINER, "--gpus", "all", "--ipc=host", "--network", "host",
+           "--memory", MEM_CAP, "--memory-swap", MEM_CAP,
+           "-v", f"{ROOT}:/work", "-e", "TORCH_CUDA_ARCH_LIST=12.1a", "-e", "PYTHONUNBUFFERED=1",
+           "-e", "PYTHONPATH=/work/exllamav3:/work", "-e", "PYTHONDONTWRITEBYTECODE=1"]
+    # encode farm: remote worker GPUs, re-read from farm.env before every run (edit without restart)
+    fenv = os.path.join(ROOT, "farm.env")
+    if os.path.exists(fenv):
+        for line in open(fenv):
+            line = line.strip()
+            if line and not line.startswith("#") and "=" in line:
+                cmd += ["-e", line]
+                log(f"  farm env: {line}")
+    cmd += ["--entrypoint", "python3", IMAGE, "/work/exllamav3/convert.py", "-w", "/work/work"]
+    if first:
+        cmd += ["-i", "/work/src", "-o", "/work/out", "-rcp", "/work/recipe.yaml",
+                "-b", str(EXPERT_BITS), "-hb", "16", "-mb", "16", "-cb", "mcg", "-cpi", "0", "-d", "0"]
+    else:
+        cmd += ["-r"]
+    if max_module is not None:
+        cmd += ["--max_module", str(max_module)]
+    log(f"RUN {tag}: {' '.join(cmd[-8:])} -> {logf}")
+    with open(logf, "w") as lf:
+        lf.write(" ".join(cmd) + "\n")
+        lf.flush()
+        rc = subprocess.call(cmd, stdout=lf, stderr=subprocess.STDOUT)
+    log(f"RUN {tag} exit={rc}")
+    return rc
+
+def ensure_present(shards):
+    while True:
+        missing = [s for s in sorted(shards) if not os.path.exists(os.path.join(SRC, s))]
+        if not missing:
+            return
+        log(f"  waiting for {len(missing)} shard(s): {missing[:3]}... free={free_gb():.0f} GB")
+        for s in missing:
+            if free_gb() < MIN_FREE_GB - 20:
+                break
+            download(s)
+        time.sleep(20)
+
+
+def container_running():
+    out = subprocess.run(["docker", "ps", "-q", "-f", f"name=^{CONTAINER}$"], capture_output=True, text=True).stdout
+    return bool(out.strip())
+
+def adopt_running(st):
+    """If a previous orchestrator left a convert run going, wait for it and credit its layers from
+    the converter's own checkpoint (work/ckpt/job.json: next_module_idx = layer + 2)."""
+    if not container_running():
+        return
+    log("adopting a running convert container; waiting for it to exit ...")
+    while container_running():
+        time.sleep(30)
+    try:
+        job = json.load(open(os.path.join(WORK, "ckpt", "job.json")))
+        nmi = int(job["next_module_idx"])
+    except Exception as e:
+        log(f"adopt: cannot read job.json ({e!r}); layers will be re-derived by the next run")
+        return
+    st = load_state()
+    done_before = len(st["done_layers"])
+    for L in range(0, min(nmi - 1, NUM_LAYERS)):
+        if L not in st["done_layers"]:
+            st["done_layers"].append(L)
+    st["started"] = True
+    save_state(st)
+    log(f"adopted: next_module_idx={nmi} -> done layers {done_before} -> {len(st['done_layers'])}")
+    drop_done_experts(st)
+
+# ---------------------------------------------------------------- main
+def main():
+    os.makedirs(SRC, exist_ok=True)
+    for fn in os.listdir(STUB):
+        dst = os.path.join(SRC, fn)
+        if not os.path.exists(dst):
+            shutil.copy(os.path.join(STUB, fn), dst)
+    st = load_state()
+    current = {"layer": max(st["done_layers"], default=-1) + 1}
+    threading.Thread(target=prefetch_loop, args=(lambda: current["layer"],), daemon=True).start()
+    log(f"orchestrator start: done={len(st['done_layers'])} layers, free={free_gb():.0f} GB, "
+        f"permanent shards={len(PERMANENT)}, window={WINDOW}, layers/run={LAYERS_PER_RUN}")
+
+    while not image_ready():
+        log(f"waiting for docker image {IMAGE} ...")
+        time.sleep(60)
+
+    adopt_running(st)
+
+    while True:
+        st = load_state()
+        done = set(st["done_layers"])
+        if st.get("finished"):
+            log("already finished")
+            return
+        next_layer = 0
+        while next_layer in done:
+            next_layer += 1
+        if next_layer >= NUM_LAYERS:
+            # final run: norm, head, MTP, compile
+            ensure_present(PERMANENT)
+            rc = run_convert(None, not st["started"], "final")
+            if rc != 0:
+                log("!! final run failed; stopping (fix and restart)")
+                return
+            st["finished"] = True
+            st["mtp_done"] = True
+            save_state(st)
+            log(f"FINISHED. output at {OUT}; free={free_gb():.0f} GB")
+            return
+        # first run: embed + dense layers 0..2 + first MoE layer in one go
+        if not st["started"]:
+            last = 3
+        else:
+            last = min(next_layer + LAYERS_PER_RUN - 1, NUM_LAYERS - 1)
+        current["layer"] = next_layer
+        need = set(PERMANENT)
+        for L in range(next_layer, last + 1):
+            need |= layer_shards[L]
+        ensure_present(need)
+        tag = f"L{next_layer:02d}-{last:02d}"
+        rc = run_convert(last + 1, not st["started"], tag)   # module idx = layer + 1 (embedding is 0)
+        if rc != 0:
+            log(f"!! run {tag} failed (exit {rc}); retrying once after 60 s")
+            time.sleep(60)
+            rc = run_convert(last + 1, False, tag + "-retry")
+            if rc != 0:
+                log("!! run failed twice; stopping. Inspect runs/ logs, then restart orchestrator.")
+                return
+        st = load_state()
+        st["started"] = True
+        for L in range(next_layer, last + 1):
+            if L not in st["done_layers"]:
+                st["done_layers"].append(L)
+        save_state(st)
+        current["layer"] = last + 1
+        drop_done_experts(st)
+        log(f"progress: {len(st['done_layers'])}/{NUM_LAYERS} layers, free={free_gb():.0f} GB")
+
+if __name__ == "__main__":
+    main()

quant/patch_aarch64.py

@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+"""Make exllamav3 1.4.5's C++/CUDA extension build on aarch64 (GB10 / DGX Spark).
+
+x86-only pieces (AVX all-reduce, AVX CPU MoE offload for the mul1 codebook, x86 pause
+intrinsics) are wrapped in architecture guards with throwing stubs. None of these paths are used
+for GPU quantization or GPU inference.  Idempotent.
+"""
+import re, sys, os
+
+root = sys.argv[1] if len(sys.argv) > 1 else "exllamav3/exllamav3_ext"
+X86 = "defined(__x86_64__) || defined(__i386__)"
+PAUSE_MACRO = """
+#if defined(__x86_64__) || defined(__i386__)
+#define EXL3_CPU_PAUSE() __builtin_ia32_pause()
+#else
+#define EXL3_CPU_PAUSE() __asm__ __volatile__("yield" ::: "memory")
+#endif
+"""
+
+def wrap(path, stub_body):
+    s = open(path).read()
+    if "EXL3_AARCH64_STUB" in s:
+        print("already:", path); return
+    s = f"#if {X86}\n" + s + f"\n#else  /* EXL3_AARCH64_STUB */\n{stub_body}\n#endif\n"
+    open(path, "w").write(s); print("wrapped:", path)
+
+def decls_from_header(path):
+    """Extract non-inline, non-template free-function declarations `ret name(args);`."""
+    s = open(path).read()
+    s = re.sub(r"//.*", "", s)
+    s = re.sub(r"/\*.*?\*/", "", s, flags=re.S)
+    out = []
+    for m in re.finditer(r"^\s*((?:[A-Za-z_][\w:<>\*&\s]*?)\s+\**\s*([A-Za-z_]\w*)\s*\(([^;{}]*?)\)\s*;)", s, flags=re.M):
+        full, name, args = m.group(1), m.group(2), m.group(3)
+        head = full[: full.index(name)].strip()
+        if head.startswith(("inline", "template", "static", "typedef", "return", "struct", "class")):
+            continue
+        if not head:
+            continue
+        out.append((head, name, " ".join(args.split())))
+    return out
+
+def stubs(header, exclude=()):
+    lines = [f'#include "{os.path.basename(header)}"', "#include <stdexcept>"]
+    for ret, name, args in decls_from_header(header):
+        if name in exclude:
+            continue
+        body = 'throw std::runtime_error("' + name + ' unavailable on aarch64");'
+        lines.append(f"{ret} {name}({args}) {{ {body} }}")
+    return "\n".join(lines)
+
+# 1. AVX all-reduce implementations
+h512 = os.path.join(root, "parallel/all_reduce_cpu_avx512.h")
+h2 = os.path.join(root, "parallel/all_reduce_cpu_avx2.h")
+wrap(os.path.join(root, "parallel/all_reduce_cpu_avx512.cpp"),
+     stubs(h512, exclude=("is_avx512_supported",)))
+wrap(os.path.join(root, "parallel/all_reduce_cpu_avx2.cpp"),
+     '#include "all_reduce_cpu_avx512.h"\n' + stubs(h2))
+
+# 2. CPU MoE (mul1) offload kernels
+hm = os.path.join(root, "cpu/moe_mul1.h")
+wrap(os.path.join(root, "cpu/moe_mul1.cpp"), stubs(hm))
+
+# 3. x86 pause intrinsic in host code of .cu files
+for f in ("parallel/all_reduce_cpu.cu", "cpu/moe_handoff.cu"):
+    p = os.path.join(root, f)
+    s = open(p).read()
+    if "EXL3_CPU_PAUSE" in s:
+        print("already:", p); continue
+    s = s.replace("__builtin_ia32_pause()", "EXL3_CPU_PAUSE()")
+    # insert macro after the last #include at file top
+    idx = 0
+    for m in re.finditer(r"^#include.*$", s, flags=re.M):
+        idx = m.end()
+    s = s[:idx] + "\n" + PAUSE_MACRO + s[idx:]
+    open(p, "w").write(s); print("pause-patched:", p)
+print("stub decls:", [n for _, n, _ in decls_from_header(hm)])

quant/patch_farm.py

@@ -0,0 +1,122 @@
+#!/usr/bin/env python3
+"""Patch exllamav3/conversion/convert_model.py so expert encoding can be farmed to remote GPUs.
+
+EXL3_REMOTE_WORKERS="host:port,host:port" adds one work thread per remote worker to the
+per-layer parallel quantization; each thread ships its groups (weights + finalized Hessians)
+via exl3_farm.quantize_remote and rebuilds LinearEXL3 from the returned tensors. Falls back to
+local encoding for a group if the remote call fails. Idempotent.
+"""
+import sys, re
+
+p = sys.argv[1]
+s = open(p).read()
+if "EXL3_REMOTE_WORKERS" in s:
+    print("already patched")
+    sys.exit(0)
+
+# 1. remote work path inside quantize_linears_parallel
+old_thread = "    def work_thread_(device_idx, dev_groups):\n        global curr_progress\n\n        with torch.inference_mode():\n"
+assert old_thread in s
+new_thread = '''    def work_thread_(device_idx, dev_groups):
+        global curr_progress
+
+        if isinstance(device_idx, str) and device_idx.startswith("remote:"):
+            _remote_work_thread(device_idx, dev_groups)
+            return
+
+        with torch.inference_mode():
+'''
+s = s.replace(old_thread, new_thread, 1)
+
+remote_fn = '''
+    def _remote_work_thread(device_idx, dev_groups):
+        global curr_progress
+        from exl3_farm import quantize_remote
+        from exllamav3.modules.quant.exl3_lib.quantize import finalize_capture_H
+        host, port = device_idx[len("remote:"):].rsplit(":", 1)
+        port = int(port)
+        local_dev = torch.device([d for d in devices if not isinstance(d, str)][0])
+        t0 = time.time()
+        with torch.inference_mode():
+            for group in dev_groups:
+                quant_args_list = [make_quant_args(args, idx, strategy[l.key], [0]) for l in group]
+                H_datas = [capture_H[l.qmap] if state else l.init_H_data(False) for l in group]
+                for hd, qa in zip(H_datas, quant_args_list):
+                    torch.manual_seed(qa["seed"])
+                    finalize_capture_H(hd, qa, False)
+                weights = [l.inner.get_weight_tensor() for l in group]
+                biases = [l.inner.get_bias_tensor() for l in group]
+                try:
+                    tq = time.time()
+                    results, quant_args_list = quantize_remote(host, port, weights, H_datas, quant_args_list)
+                    tstr = f"  [remote {host} {(time.time() - tq) / len(group):4.2f} s]"
+                except Exception as e:
+                    print(f" !! remote {host}:{port} failed ({e!r}); encoding group locally", flush = True)
+                    for l in group:
+                        l.inner.swap_cpu()
+                    if len(group) > 1:
+                        quant_args_list = [make_quant_args(args, idx, strategy[l.key], [local_dev.index or 0]) for l in group]
+                        proxy_errs = convert_exl3_group(group, H_datas, quant_args_list)
+                        for l, qa, pe in zip(group, quant_args_list, proxy_errs):
+                            l.inner.swap_cpu()
+                            print_quantized_linear(config, l, qa, pe, "  [local fallback]")
+                            with progress_lock:
+                                curr_progress += 1
+                    else:
+                        l = group[0]
+                        qa = make_quant_args(args, idx, strategy[l.key], [local_dev.index or 0])
+                        pe = l.convert_exl3(H_datas[0], quant_args = qa, verbose = False, save_reg = False,
+                                            override_swap_device = local_dev)
+                        l.inner.swap_cpu()
+                        print_quantized_linear(config, l, qa, pe, "  [local fallback]")
+                        with progress_lock:
+                            curr_progress += 1
+                    continue
+                for l, bias, qa, (pe, out) in zip(group, biases, quant_args_list, results):
+                    out = {k: v.to(local_dev) for k, v in out.items()}
+                    l.inner = LinearEXL3(
+                        l.config, l.in_features, l.out_features,
+                        out.get("scale"), out.get("su"), out.get("sv"), out.get("suh"), out.get("svh"),
+                        out.get("trellis"), out.get("mcg"), out.get("mul1"),
+                        bias, l.out_dtype, key = l.key
+                    )
+                    l.inner.swap_cpu()
+                    print_quantized_linear(config, l, qa, pe, tstr)
+                    with progress_lock:
+                        curr_progress += 1
+        print(f" -- remote {host}: {sum(len(g) for g in dev_groups)} tensors in {time.time() - t0:.0f} s", flush = True)
+
+'''
+anchor = "    # Launch\n    threads = []\n"
+assert anchor in s
+s = s.replace(anchor, remote_fn + anchor, 1)
+
+# 2. dispatch: farm the <=8-bit linears, keep 16-bit ones on the single path
+old_dispatch = '''            if (
+                len(linears) >= len(devices) and
+                all(b <= 8 for _, b in strategy.items())
+            ):
+                quantize_linears_parallel(args, linears, config, strategy, idx, devices, eff_ratios("quant_thread"), capture_H, state)
+            else:
+                quantize_linears_single(args, linears, config, strategy, idx, devices, eff_ratios("quant_tiles"), capture_H, state)
+'''
+assert old_dispatch in s
+new_dispatch = '''            _remotes = [f"remote:{w.strip()}" for w in os.environ.get("EXL3_REMOTE_WORKERS", "").split(",") if w.strip()]
+            _q_linears = [l for l in linears if strategy[l.key] <= 8]
+            _u_linears = [l for l in linears if strategy[l.key] > 8]
+            if _remotes and len(_q_linears) >= 2 * (len(devices) + len(_remotes)):
+                if _u_linears:
+                    quantize_linears_single(args, _u_linears, config, strategy, idx, devices, eff_ratios("quant_tiles"), capture_H, state)
+                print(f" -- Farming {len(_q_linears)} tensors over {devices + _remotes}", flush = True)
+                quantize_linears_parallel(args, _q_linears, config, strategy, idx, devices + _remotes, None, capture_H, state)
+            elif (
+                len(linears) >= len(devices) and
+                all(b <= 8 for _, b in strategy.items())
+            ):
+                quantize_linears_parallel(args, linears, config, strategy, idx, devices, eff_ratios("quant_thread"), capture_H, state)
+            else:
+                quantize_linears_single(args, linears, config, strategy, idx, devices, eff_ratios("quant_tiles"), capture_H, state)
+'''
+s = s.replace(old_dispatch, new_dispatch, 1)
+open(p, "w").write(s)
+print("patched", p)

quant/start-orch.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+cd ~/glm53-full-exl3 && nohup python3 orchestrate.py >> orchestrate.out 2>&1 &

serving/Dockerfile.dcp

@@ -0,0 +1,21 @@
+# keys GLM-5.3 FULL EXL3 (3-bit mcg routed experts) serving image = Mia base + vertical integration:
+#   * keys-DCP port of the SM120 sparse-MLA backend (KV sharded across TP ranks -> 1M ctx on 4 GB10)
+#   * MLA layer DCP combine fix for sparse all-MQA backends (no MHA prefill path)
+#   * ABLIT load-time o_proj transplant hook for the DeepseekV2 model classes (derisk donors from
+#     make_derisk_donor.py mounted at /opt/glm53/ablit/transplant; ABLIT=1 to enable)
+# Build on any GB10 (context = this directory):  docker build -f Dockerfile.dcp -t glm53-exl3-tp4:dcp .
+ARG BASE=ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3
+FROM ${BASE}
+ARG VLLM=/usr/local/lib/python3.12/dist-packages/vllm
+COPY sm120_dcp.py ${VLLM}/v1/attention/backends/mla/flashinfer_mla_sparse_sm120.py
+COPY mla_attn_dcp.py ${VLLM}/model_executor/layers/attention/mla_attention.py
+COPY patch_ablit_dsv2.py /opt/glm53/patch_ablit_dsv2.py
+COPY ablit_runtime.py /opt/glm53/ablit_runtime.py
+RUN python3 /opt/glm53/patch_ablit_dsv2.py /opt/glm53/ablit_runtime.py && python3 - <<'EOF'
+from vllm.v1.attention.backends.mla.flashinfer_mla_sparse_sm120 import FlashInferMLASparseSM120Impl as I
+assert I.can_return_lse_for_decode is True
+import vllm.model_executor.layers.attention.mla_attention as m
+assert "keys-DCP" in open(m.__file__).read()
+print("keys image OK: DCP backend + layer fix + ablit hook; lse_base_on_e =", I.lse_base_on_e)
+EOF
+LABEL keys.glm53.full.exl3="DCP sm120 + mla combine fix + ablit dsv2 hook, 2026-09-01"

serving/Dockerfile.keys

@@ -0,0 +1,2 @@
+FROM glm53-exl3-tp4:dcp-mul1-v145
+RUN f=/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/sparse_attn_indexer.py && sed -i "s/use_persistent_topk = current_platform.is_cuda() and topk_tokens in (/use_persistent_topk = False and current_platform.is_cuda() and topk_tokens in (  # keys: GB10 48-SM oversubscribe/" $f && grep -c "keys: GB10 48-SM" $f && rm -rf /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/__pycache__/sparse_attn_indexer*

serving/Dockerfile.keys3

@@ -0,0 +1,3 @@
+FROM glm53-exl3-tp4:keys
+COPY exl3.py /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/exl3.py
+RUN rm -rf /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/__pycache__

serving/Dockerfile.mul1

@@ -0,0 +1,2 @@
+FROM glm53-exl3-tp4:dcp
+COPY exl3.py /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/exl3.py

serving/Dockerfile.v145

@@ -0,0 +1,6 @@
+FROM glm53-exl3-tp4:dcp-mul1
+ARG DP=/usr/local/lib/python3.12/dist-packages
+RUN rm -rf $DP/exllamav3 $DP/exllamav3-0.0.43.dist-info $DP/exllamav3_ext*.so
+COPY --from=glm53-exl3-quant:1.4.5 $DP/exllamav3 $DP/exllamav3
+COPY --from=glm53-exl3-quant:1.4.5 $DP/exllamav3-1.4.5.dist-info $DP/exllamav3-1.4.5.dist-info
+COPY --from=glm53-exl3-quant:1.4.5 $DP/exllamav3_ext.cpython-312-aarch64-linux-gnu.so $DP/exllamav3_ext.cpython-312-aarch64-linux-gnu.so

serving/ablit_runtime.py

@@ -0,0 +1,460 @@
+# SPDX-License-Identifier: MIT
+"""GLM-5.3-Flash o_proj abliteration (ABLIT) — load-time refusal-direction edit.
+
+Applies the classic weight orthogonalization to every ``self_attn.o_proj`` in
+the configured layer range:
+
+    W' = (I - alpha * r r^T / ||r||^2) W
+
+``r`` is the refusal direction in the residual/output space of o_proj
+(hidden_size=4096). Components of the attention output orthogonal to ``r``
+are preserved exactly; the component along ``r`` is scaled by ``1 - alpha``
+(alpha > 1 over-projects, which is what the published recipe uses).
+
+METHOD (ABLIT_METHOD=auto | transplant | proj):
+  transplant — replace the stock o_proj weights L15-45 (incl. the checkpoint
+    MTP block's, layer 45) with the donor tensors fetched by
+    ablit/fetch_transplant.py into ablit/transplant/ (raw bf16 + MANIFEST.json).
+    This is the published "dealign-oproj-transplant" edit — a byte-copy from
+    dealignai/GLM-5.3-Flash-UNCENSORED-NVFP4 into the stock body. Direction
+    orthogonalization does NOT work on this model (the published directions
+    are statistically random vs stock o_proj — measured), so transplant is
+    the only effective mode; "auto" prefers it when the tensors are present.
+  proj — the classic orthogonalization below. Kept for custom directions.
+
+TP notes: o_proj is a RowParallelLinear — vLLM shards the *input* dim, the
+4096 output rows are replicated on every rank. The edit only touches rows,
+so each rank applies the identical formula to its own shard and the
+post-allreduce result stays consistent. No collective is needed.
+
+The edit runs at the end of ``Glm5NextModel.load_weights`` / ``Glm5NextMTP.
+load_weights`` (installed by overlay/patch_ablit.py). o_proj stays native
+BF16 in this serve (attn is unquantized), so the parameter is final at that
+point — quantized-expert post-processing never touches it, and CUDA graph
+capture happens later.
+
+Artifacts (shipped in ``ablit/`` and mounted at /opt/glm53/ablit):
+  LAYER_MAP.json                              layer/shard map + published recipe
+  refusal_direction_glm53_dealign_late.pt     published dealign direction
+  refusal_direction_glm53_bf_oproj.pt         blackfrost direction (alpha_ref 3.0)
+
+Source: drowzeys/keys-GLM-5.3-Flash-NVFP4-ablit-l15-45-anchorstock
+(published method "dealign-oproj-transplant": layers 15-45 edited, 0-14 kept
+as stock safety anchors, MTP block included).
+
+Env knobs (see .env.example):
+  ABLIT=1                  enable (default off — hook is a no-op)
+  ABLIT_DIR                artifact dir (default /opt/glm53/ablit)
+  ABLIT_METHOD             auto | transplant | proj (default auto: transplant
+                           when ablit/transplant/ is populated, else proj)
+  ABLIT_DIRECTION          dealign | bf_oproj | /path/to/dir.pt (default dealign)
+  ABLIT_LAYERS             inclusive ranges, e.g. "15-45" or "15,17-19"
+  ABLIT_ALPHA              projection scale (default 3.0)
+  ABLIT_INCLUDE_MTP        also edit the checkpoint MTP block when it exists
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import os
+import re
+from pathlib import Path
+from typing import Any
+
+import torch
+
+def _tp_world() -> int:
+    try:
+        from vllm.distributed import get_tensor_model_parallel_world_size
+        return get_tensor_model_parallel_world_size()
+    except Exception:  # standalone / tests
+        return 1
+
+
+def _tp_rank() -> int:
+    try:
+        from vllm.distributed import get_tensor_model_parallel_rank
+        return get_tensor_model_parallel_rank()
+    except Exception:  # standalone / tests
+        return 0
+
+
+try:  # inside the vLLM image
+    from vllm.logger import init_logger
+
+    logger = init_logger(__name__)
+except Exception:  # standalone (tests)
+    import logging
+
+    logger = logging.getLogger("glm53_ablit")
+
+DEFAULT_ABLIT_DIR = "/opt/glm53/ablit"
+TRANSPLANT_SUBDIR = "transplant"
+
+DIRECTION_FILES = {
+    "dealign": "refusal_direction_glm53_dealign_late.pt",
+    "bf_oproj": "refusal_direction_glm53_bf_oproj.pt",
+}
+
+# paths inside the hooked model that carry an o_proj to edit
+_LAYER_O_PROJ_RE = re.compile(r"(?:^|\.)layers\.(\d+)\.self_attn\.o_proj$")
+_MTP_O_PROJ_RE = re.compile(r"(?:^|\.)layers\.(\d+)\.mtp_block\.self_attn\.o_proj$")
+
+
+class AblitError(RuntimeError):
+    """ABLIT was explicitly enabled but the edit cannot be applied."""
+
+
+def env_flag(name: str, default: bool = False) -> bool:
+    val = os.environ.get(name)
+    if val is None or val == "":
+        return default
+    return val.strip().lower() in ("1", "true", "yes", "on")
+
+
+def parse_layers(spec: str) -> list[int]:
+    """Parse "15-45" / "15,17-19" into a sorted list of layer indices."""
+    layers: set[int] = set()
+    for part in spec.split(","):
+        part = part.strip()
+        if not part:
+            continue
+        if "-" in part:
+            lo_s, hi_s = part.split("-", 1)
+            try:
+                lo, hi = int(lo_s), int(hi_s)
+            except ValueError as exc:
+                raise AblitError(f"ABLIT_LAYERS: bad range {part!r}") from exc
+            if hi < lo:
+                raise AblitError(f"ABLIT_LAYERS: inverted range {part!r}")
+            layers.update(range(lo, hi + 1))
+        else:
+            try:
+                layers.add(int(part))
+            except ValueError as exc:
+                raise AblitError(f"ABLIT_LAYERS: bad index {part!r}") from exc
+    if not layers:
+        raise AblitError("ABLIT_LAYERS resolved to an empty set")
+    return sorted(layers)
+
+
+def resolve_direction_path(ablit_dir: str, direction: str) -> Path:
+    if "/" in direction:  # explicit path to a .pt
+        return Path(direction)
+    fname = DIRECTION_FILES.get(direction)
+    if fname is None:
+        raise AblitError(
+            f"ABLIT_DIRECTION={direction!r} not one of "
+            f"{sorted(DIRECTION_FILES)} (or a .pt path)"
+        )
+    return Path(ablit_dir) / fname
+
+
+def load_direction(path: Path) -> torch.Tensor:
+    try:
+        obj = torch.load(str(path), map_location="cpu", weights_only=True)
+    except Exception as exc:
+        raise AblitError(f"cannot load ablit direction {path}: {exc}") from exc
+    if not isinstance(obj, dict) or "directions" not in obj:
+        raise AblitError(f"{path}: expected a dict with 'directions'")
+    r = obj["directions"]
+    if not torch.is_tensor(r):
+        raise AblitError(f"{path}: 'directions' must be a tensor, got {type(r)}")
+    if r.dim() == 2 and r.shape[0] == 1:  # some exports store a [1, N] row
+        r = r.squeeze(0)
+    if r.dim() != 1:
+        raise AblitError(f"{path}: 'directions' must be 1-D (or [1, N]), got {tuple(r.shape)}")
+    r = r.to(torch.float32)
+    norm = r.norm()
+    if not torch.isfinite(norm) or float(norm) <= 0:
+        raise AblitError(f"{path}: direction has non-positive/invalid norm {norm}")
+    return r
+
+
+def apply_to_o_proj(mod: Any, r: torch.Tensor, alpha: float) -> dict[str, Any]:
+    """Orthogonalize one o_proj module in place. Returns a report dict."""
+    weight = getattr(mod, "weight", None)
+    if weight is None or not torch.is_tensor(weight) or weight.dim() != 2:
+        return {"edited": False, "reason": "no 2-D .weight"}
+    if weight.shape[0] != r.numel():
+        return {
+            "edited": False,
+            "reason": f"out_features={weight.shape[0]} != direction dim {r.numel()}",
+        }
+    r_dev = r.to(device=weight.device, dtype=torch.float32)
+    with torch.no_grad():
+        w32 = weight.data.to(torch.float32)
+        # W' = W - alpha * outer(r, r @ W) / ||r||^2
+        rw = r_dev @ w32  # [in_local]
+        delta = alpha * torch.outer(r_dev, rw) / float(r_dev @ r_dev)
+        w32.sub_(delta)
+        weight.data.copy_(w32.to(weight.dtype))
+        residual = float((r_dev @ weight.data.to(torch.float32)).abs().max())
+    return {
+        "edited": True,
+        "shape": tuple(weight.shape),
+        "residual_max": residual,
+    }
+
+
+def walk_o_proj(model: Any) -> list[tuple[str, int | None, Any]]:
+    """Collect (name, layer_idx or None-if-pure-mtp, module) o_proj candidates."""
+    found: list[tuple[str, int | None, Any]] = []
+    seen: set[int] = set()
+    for name, mod in model.named_modules():
+        if id(mod) in seen:
+            continue
+        m = _LAYER_O_PROJ_RE.search(name)
+        if m is not None:
+            seen.add(id(mod))
+            found.append((name, int(m.group(1)), mod))
+            continue
+        m = _MTP_O_PROJ_RE.search(name)
+        if m is not None:
+            seen.add(id(mod))
+            found.append((name, int(m.group(1)), mod))
+    return found
+
+
+def unwrap_text_model(model: Any) -> Any:
+    """Accept the multimodal wrapper and hand back the text model."""
+    lm = getattr(model, "language_model", None)
+    if lm is not None:
+        inner = getattr(lm, "model", None)
+        if inner is not None:
+            return inner
+        return lm
+    return model
+
+
+def apply_ablit(
+    model: Any,
+    r: torch.Tensor,
+    layers: list[int],
+    alpha: float,
+    include_mtp: bool,
+) -> dict[str, Any]:
+    """Edit every configured o_proj under ``model``. Fails hard on surprises."""
+    text_model = unwrap_text_model(model)
+    candidates = walk_o_proj(text_model)
+    want = set(layers)
+    report: dict[str, Any] = {
+        "edited_layers": [],
+        "skipped": [],
+        "mtp_edited": False,
+    }
+    seen_ids: set[int] = set()
+    for name, idx, mod in candidates:
+        if id(mod) in seen_ids:
+            continue
+        seen_ids.add(id(mod))
+        is_mtp = _MTP_O_PROJ_RE.search(name) is not None
+        if is_mtp:
+            if not include_mtp:
+                report["skipped"].append({"name": name, "reason": "include_mtp=0"})
+                continue
+            if idx not in want:
+                report["skipped"].append(
+                    {"name": name, "reason": f"layer {idx} not in ABLIT_LAYERS"}
+                )
+                continue
+        else:
+            if idx not in want:
+                continue
+        rep = apply_to_o_proj(mod, r, alpha)
+        if rep.get("edited"):
+            if is_mtp:
+                report["mtp_edited"] = True
+            else:
+                report["edited_layers"].append(idx)
+            logger.info(
+                "ablit: orthogonalized %s shape=%s residual_max=%.3e",
+                name,
+                rep.get("shape"),
+                rep.get("residual_max", float("nan")),
+            )
+        else:
+            report["skipped"].append({"name": name, "reason": rep.get("reason")})
+            logger.warning("ablit: skipped %s: %s", name, rep.get("reason"))
+    report["edited_layers"].sort()
+    return report
+
+
+def load_transplant_tensors(
+    ablit_dir: str, layers: list[int]
+) -> dict[int, torch.Tensor]:
+    """Load donor o_proj tensors (full, unsharded) from ablit/transplant/."""
+    tdir = Path(ablit_dir) / TRANSPLANT_SUBDIR
+    manifest_path = tdir / "MANIFEST.json"
+    if not manifest_path.is_file():
+        raise AblitError(f"ABLIT_METHOD=transplant but {manifest_path} is missing "
+                         "— run ablit/fetch_transplant.py first")
+    manifest = json.loads(manifest_path.read_text())
+    meta = {int(k): v for k, v in (manifest.get("layers") or manifest.get("tensors") or {}).items()}
+    out: dict[int, torch.Tensor] = {}
+    for L in layers:
+        if L not in meta:
+            continue
+        info = meta[L]
+        path = tdir / f"L{L}.bin"
+        if not path.is_file():
+            raise AblitError(f"transplant manifest lists L{L} but {path} is missing")
+        raw = path.read_bytes()
+        if len(raw) != int(info["nbytes"]):
+            raise AblitError(
+                f"transplant L{L}: expected {info['nbytes']} bytes, got {len(raw)} "
+                "— re-run ablit/fetch_transplant.py")
+        if info["dtype"] != "BF16":
+            raise AblitError(f"transplant L{L}: unsupported dtype {info['dtype']}")
+        t = torch.frombuffer(bytearray(raw), dtype=torch.bfloat16).reshape(
+            tuple(info["shape"]))
+        out[L] = t
+    if not out:
+        raise AblitError(
+            f"no transplant tensors under {tdir} cover ABLIT_LAYERS — "
+            "run ablit/fetch_transplant.py first")
+    return out
+
+
+def apply_transplant(
+    model: Any,
+    donors: dict[int, torch.Tensor],
+    layers: list[int],
+    include_mtp: bool,
+) -> dict[str, Any]:
+    """Replace stock o_proj weights with the donor tensors (per-TP shard)."""
+    text_model = unwrap_text_model(model)
+    candidates = walk_o_proj(text_model)
+    want = set(layers)
+    world, rank = _tp_world(), _tp_rank()
+    report: dict[str, Any] = {"edited_layers": [], "skipped": [], "mtp_edited": False,
+                              "deltas": {}}
+    seen_ids: set[int] = set()
+    for name, idx, mod in candidates:
+        if id(mod) in seen_ids:
+            continue
+        seen_ids.add(id(mod))
+        is_mtp = _MTP_O_PROJ_RE.search(name) is not None
+        if is_mtp and not include_mtp:
+            report["skipped"].append({"name": name, "reason": "include_mtp=0"})
+            continue
+        if idx not in want:
+            continue
+        if idx not in donors:
+            raise AblitError(
+                f"ABLIT_METHOD=transplant has no donor tensor for layer {idx} "
+                "— fetch it with ablit/fetch_transplant.py")
+        donor = donors[idx]
+        weight = getattr(mod, "weight", None)
+        if weight is None or not torch.is_tensor(weight) or weight.dim() != 2:
+            raise AblitError(f"ablit transplant: {name} has no 2-D .weight")
+        full_in = donor.shape[1]
+        local_in = weight.shape[1]
+        if donor.shape[0] != weight.shape[0] or full_in != local_in * world:
+            raise AblitError(
+                f"ablit transplant: {name} shape {tuple(weight.shape)} does not "
+                f"match donor {tuple(donor.shape)} at TP={world}")
+        shard = donor if world == 1 else donor[:, rank * local_in:(rank + 1) * local_in]
+        # Donors come from torch.frombuffer on CPU; o_proj is already on the
+        # worker GPU. Compare and copy on weight's device (proj path already
+        # does r.to(weight.device); transplant forgot).
+        shard = shard.to(device=weight.device, dtype=weight.dtype)
+        with torch.no_grad():
+            w32 = weight.data.float()
+            d32 = shard.float()
+            rel_l2 = float((d32 - w32).norm() / w32.norm().clamp_min(1e-9))
+            mean_rel = float((d32 - w32).abs().mean() / w32.abs().mean().clamp_min(1e-9))
+            weight.data.copy_(shard)
+        report["edited_layers"].append(idx)
+        report["deltas"][idx] = {"rel_l2": rel_l2, "mean_rel": mean_rel}
+        if is_mtp:
+            report["mtp_edited"] = True
+        logger.info(
+            "ablit: transplanted %s (donor L%d) shape=%s rel_l2=%.4f mean_rel=%.4f",
+            name, idx, tuple(weight.shape), rel_l2, mean_rel)
+    report["edited_layers"].sort()
+    return report
+
+
+def maybe_apply(model: Any) -> dict[str, Any] | None:
+    """Hook entrypoint. No-op unless ABLIT=1; raises AblitError if enabled
+    but the recipe cannot be honored (fail loud beats silent stock weights)."""
+    if not env_flag("ABLIT", False):
+        return None
+
+    ablit_dir = os.environ.get("ABLIT_DIR") or DEFAULT_ABLIT_DIR
+    layers_spec = os.environ.get("ABLIT_LAYERS") or "15-45"
+    include_mtp = env_flag("ABLIT_INCLUDE_MTP", True)
+    method = (os.environ.get("ABLIT_METHOD") or "auto").strip().lower()
+    layers = parse_layers(layers_spec)
+
+    if method in ("auto", "transplant"):
+        tdir = Path(ablit_dir) / TRANSPLANT_SUBDIR
+        have_transplant = (tdir / "MANIFEST.json").is_file() and (
+            method == "transplant"
+            or any((tdir / f"L{L}.bin").is_file() for L in layers)
+        )
+        if have_transplant:
+            if method == "auto":
+                logger.info("ablit: ABLIT_METHOD=auto -> transplant "
+                            "(ablit/transplant/ present; direction proj does "
+                            "not align with stock o_proj on this model)")
+            donors = load_transplant_tensors(ablit_dir, layers)
+            report = apply_transplant(model, donors, layers, include_mtp)
+            if not report["edited_layers"] and not report["mtp_edited"]:
+                raise AblitError(
+                    "ABLIT=1 transplant matched no o_proj — ABLIT_LAYERS="
+                    f"{layers_spec} matched nothing under this model")
+            deltas = list(report["deltas"].values())
+            logger.info(
+                "ablit: ON method=transplant layers=%s edited=%s mtp=%s "
+                "mean rel_l2=%.4f (skipped=%d) — early safety-anchor layers "
+                "stay stock",
+                layers_spec,
+                report["edited_layers"],
+                report["mtp_edited"],
+                sum(d["rel_l2"] for d in deltas) / max(len(deltas), 1),
+                len(report["skipped"]),
+            )
+            return report
+        if method == "transplant":
+            raise AblitError(
+                "ABLIT_METHOD=transplant but ablit/transplant/ is missing or "
+                "empty — run: python3 ablit/fetch_transplant.py")
+        logger.info("ablit: ABLIT_METHOD=auto -> proj (no ablit/transplant/ "
+                    "tensors found — falling back to direction orthogonalization)")
+
+    # ABLIT_METHOD=proj (or auto fallback): direction orthogonalization
+    direction = os.environ.get("ABLIT_DIRECTION") or "dealign"
+    try:
+        alpha = float(os.environ.get("ABLIT_ALPHA") or "3.0")
+    except ValueError as exc:
+        raise AblitError(f"ABLIT_ALPHA is not a number: {exc}") from exc
+    if not math.isfinite(alpha) or alpha <= 0:
+        raise AblitError(f"ABLIT_ALPHA must be a positive finite number, got {alpha}")
+
+    path = resolve_direction_path(ablit_dir, direction)
+    if not path.is_file():
+        raise AblitError(f"ABLIT=1 but direction file missing: {path}")
+    r = load_direction(path)
+
+    report = apply_ablit(model, r, layers, alpha, include_mtp)
+
+    if not report["edited_layers"] and not report["mtp_edited"]:
+        raise AblitError(
+            "ABLIT=1 but no o_proj was edited — ABLIT_LAYERS="
+            f"{layers_spec} matched nothing under this model"
+        )
+    logger.info(
+        "ablit: ON method=proj direction=%s (%s) alpha=%s layers=%s edited=%s "
+        "mtp=%s (skipped=%d) — early safety-anchor layers stay stock",
+        direction,
+        path.name,
+        alpha,
+        layers_spec,
+        report["edited_layers"],
+        report["mtp_edited"],
+        len(report["skipped"]),
+    )
+    return report

serving/donor_diff.py

@@ -0,0 +1,14 @@
+import json, torch, os, sys
+from safetensors import safe_open
+M="/models/GLM-5.3-EXL3-3bpw"; D="/models/glm53-ablit-donor-derisked"
+idx=json.load(open(f"{M}/model.safetensors.index.json"))["weight_map"]
+man=json.load(open(f"{D}/MANIFEST.json"))["layers"]
+print("layer rel_l2 cos donor_bits")
+for L in range(79):
+    key=f"model.layers.{L}.self_attn.o_proj.weight"
+    if key not in idx: print(L,"no-stock"); continue
+    with safe_open(f"{M}/{idx[key]}","pt") as f: w=f.get_tensor(key).float()
+    info=man[str(L)]; raw=open(f"{D}/L{L}.bin","rb").read()
+    d=torch.frombuffer(bytearray(raw),dtype=torch.bfloat16).view(info["shape"]).float()
+    rel=((d-w).norm()/w.norm()).item(); cos=torch.nn.functional.cosine_similarity(d.flatten(),w.flatten(),dim=0).item()
+    print(f"{L:2d} {rel:.4f} {cos:.5f} {info.get('bits')}", flush=True)

serving/env.tp4.example

@@ -0,0 +1,47 @@
+# GLM-5.3 FULL EXL3 3bpw mcg — 4x DGX Spark TP=4 (+DCP=4 for 1M) — DRAFT for ~/glm53-mia/.env.tp4 on .1
+# ranks: .1 head, .2 / .3 / .5 workers (all on the 10.100.10.x CX7 fabric, same NIC names)
+HEAD_IP=10.100.10.1
+WORKER_IP=10.100.10.2
+WORKER2_IP=10.100.10.3
+WORKER3_IP=10.100.10.5
+WORKER_USER=keyspark
+NCCL_CROSS_NIC=0
+HEAD_CX7_IF=enp1s0f1np1
+HEAD_CX7_IB=rocep1s0f1
+WORKER_CX7_IF=enP2p1s0f1np1
+WORKER_CX7_IB=roceP2p1s0f1
+WORKER2_CX7_IF=enp1s0f1np1
+WORKER2_CX7_IB=rocep1s0f1
+WORKER3_CX7_IF=enp1s0f1np1
+WORKER3_CX7_IB=rocep1s0f1
+NCCL_IB_GID_INDEX=3
+NCCL_DEBUG=WARN
+
+TP=4
+NNODES=4
+PORT=8888
+MASTER_PORT=29521
+CONTAINER_HEAD=glm53-exl3-tp4-head
+CONTAINER_WORKER=glm53-exl3-tp4-w1
+CONTAINER_WORKER2=glm53-exl3-tp4-w2
+CONTAINER_WORKER3=glm53-exl3-tp4-w3
+
+# model: quantized on .3 -> exported over NFS as /mnt/models-node2/<name> on every node
+LOCAL_MODEL_DIR=/mnt/models-node2/GLM-5.3-EXL3-3bpw
+MODEL=keys/GLM-5.3-EXL3-3bpw
+SERVED_MODEL_NAME=GLM-5.3-EXL3
+IMAGE=glm53-exl3-tp4:keys3
+QUANTIZATION=exl3
+KV_CACHE_DTYPE=fp8
+MAX_MODEL_LEN=1000000
+GPU_MEM_UTIL=0.82
+MAX_NUM_SEQS=8
+MAX_NUM_BATCHED_TOKENS=4096
+SPEC_METHOD=mtp
+LANGUAGE_MODEL_ONLY=1
+ENFORCE_EAGER=0
+EXTRA_ARGS="--decode-context-parallel-size 4 --kv-cache-memory-bytes 14092861440"
+LOCAL_DFLASH_DIR=/mnt/models-node2/GLM-5.3-DFlash2
+DFLASH_TOKENS=7
+GLM53_MIXED_PREFILL_CHUNK=2048
+MTP_TOKENS=3

serving/exl3.py

@@ -0,0 +1,862 @@
+# SPDX-License-Identifier: Apache-2.0
+"""EXL3/MCG trellis quantization for GLM-5.3-Flash routed experts.
+
+Checkpoint ABI (brandonmusic/GLM-5.3-Flash-tr3-4bpw):
+  quant_method=exl3, codebook=mcg, scope=glm53_routed_experts_only
+  per expert matrix: trellis (int16) + suh/svh (fp16) + mcg (int32 marker)
+
+Non-routed tensors stay native (UnquantizedLinearMethod). Experts never
+expand to a persistent BF16 weight; LinearEXL3 / exllamav3_ext runs the
+trellis GEMM. TP=2 shards gate/up column-wise and down row-wise; the MoE
+runner all-reduces the combined output.
+"""
+
+from __future__ import annotations
+
+import importlib
+import importlib.util
+import os
+import sys
+import types
+from pathlib import Path
+from typing import TYPE_CHECKING, Any
+
+import torch
+import torch.nn.functional as F
+from torch.nn.parameter import Parameter
+
+from vllm.logger import init_logger
+from vllm.model_executor.layers.fused_moe.config import FusedMoEQuantConfig
+from vllm.model_executor.layers.fused_moe.fused_moe_method_base import (
+    FusedMoEMethodBase,
+)
+from vllm.model_executor.layers.linear import LinearBase, UnquantizedLinearMethod
+from vllm.model_executor.layers.quantization.base_config import QuantizationConfig
+from vllm.model_executor.layers.quantization import register_quantization_config
+from vllm.model_executor.utils import set_weight_attrs
+
+if TYPE_CHECKING:
+    from vllm.model_executor.layers.fused_moe.routed_experts import RoutedExperts
+    from vllm.model_executor.layers.fused_moe.runner.shared_experts import (
+        SharedExperts,
+    )
+
+logger = init_logger(__name__)
+
+EXLLAMAV3_COMMIT = "c5d9c657966ffeeaa9353f0cc899f18629da4a13"
+EXLLAMAV3_VERSION = "0.0.43"
+MCG_MULTIPLIER = 0xCBAC1FED
+MCG_MARKER_SIGNED_INT32 = -877912083
+# KEYS PATCH: exllamav3 "mul1" codebook (converter default; 0x83DCD12D marker). LinearEXL3 and
+# exllamav3_ext.exl3_moe in this image already take (mcg, mul1) flag pairs per matrix.
+MUL1_MULTIPLIER = 0x83DCD12D
+MUL1_MARKER_SIGNED_INT32 = -2082680531
+EXL3_SUFFIXES = ("trellis", "suh", "svh", "mcg", "mul1")
+SWIGLU_LIMIT_DEFAULT = 10.0
+TEMP_ROWS_FUSED = 128
+MOE_ACT_SILU = 0
+# Shared fused scratch: decode is sequential across layers.
+_FUSED_TEMP_CACHE: dict[tuple, tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]] = {}
+
+
+def _narrow_tp(tensor: torch.Tensor, dim: int, tp_rank: int, tp_size: int) -> torch.Tensor:
+    if tp_size <= 1:
+        return tensor
+    size = int(tensor.shape[dim])
+    if size % tp_size:
+        raise ValueError(
+            f"EXL3 TP shard: dim {dim} size {size} is not divisible by tp={tp_size}"
+        )
+    chunk = size // tp_size
+    return tensor.narrow(dim, chunk * tp_rank, chunk).contiguous()
+
+
+def shard_exl3_col(loaded: torch.Tensor, suffix: str, tp_rank: int, tp_size: int) -> torch.Tensor:
+    """Gate/up: trellis dim 1 and svh dim 0 are column-parallel."""
+    if suffix == "trellis":
+        return _narrow_tp(loaded, 1, tp_rank, tp_size)
+    if suffix == "svh":
+        return _narrow_tp(loaded, 0, tp_rank, tp_size)
+    return loaded.contiguous()
+
+
+def shard_exl3_row(loaded: torch.Tensor, suffix: str, tp_rank: int, tp_size: int) -> torch.Tensor:
+    """Down: trellis dim 0 and suh dim 0 are row-parallel."""
+    if suffix == "trellis":
+        return _narrow_tp(loaded, 0, tp_rank, tp_size)
+    if suffix == "suh":
+        return _narrow_tp(loaded, 0, tp_rank, tp_size)
+    return loaded.contiguous()
+
+
+def _install_exllamav3_namespace() -> None:
+    """Load LinearEXL3 without running exllamav3/__init__.py (FlashAttention)."""
+    if "exllamav3.modules.quant.exl3" in sys.modules:
+        return
+    import exllamav3_ext  # noqa: F401  — compiled extension must exist
+
+    spec = importlib.util.find_spec("exllamav3")
+    if spec is None or not spec.submodule_search_locations:
+        raise RuntimeError("exllamav3 package is not installed in this image")
+    package_root = Path(list(spec.submodule_search_locations)[0])
+
+    # Stub only packages whose __init__.py pulls FlashAttention / serving extras.
+    # Leave .ext, .util, and .modules.quant as real modules so LinearEXL3 loads.
+    for name, path in (
+        ("exllamav3", package_root),
+        ("exllamav3.modules", package_root / "modules"),
+        ("exllamav3.model", package_root / "model"),
+    ):
+        if name in sys.modules:
+            continue
+        module = types.ModuleType(name)
+        module.__file__ = str(path / "__init__.py")
+        module.__package__ = name
+        module.__path__ = [str(path)]
+        sys.modules[name] = module
+
+    if "exllamav3.model.config" not in sys.modules:
+        config = types.ModuleType("exllamav3.model.config")
+        config.__file__ = str(package_root / "model/config.py")
+        config.__package__ = "exllamav3.model"
+        config.Config = type("Config", (), {})
+        # KEYS PATCH: exllamav3 >= 1.x LinearEXL3(config=None) does
+        # `from ...model.config import NullConfig` and reads
+        # config.infer_params.no_reconstruct in forward().
+        class _InferParams:
+            no_reconstruct = False
+            mgemm_K_threshold = 0
+            mgemm_n_threshold = 0
+
+        class _NullConfig:
+            def __init__(self):
+                self.infer_params = _InferParams()
+
+        config.InferParams = _InferParams
+        config.NullConfig = _NullConfig
+        sys.modules[config.__name__] = config
+
+
+def load_linear_exl3_cls():
+    _install_exllamav3_namespace()
+    return importlib.import_module("exllamav3.modules.quant.exl3").LinearEXL3
+
+
+def make_linear_exl3(
+    trellis: torch.Tensor,
+    suh: torch.Tensor,
+    svh: torch.Tensor,
+    mcg: torch.Tensor,
+    *,
+    mul1: bool = False,
+    out_dtype: torch.dtype = torch.float16,
+):
+    """Build a LinearEXL3 over already-sharded packed tensors. No BF16 expand.
+    KEYS PATCH: `mcg` is the codebook marker tensor; with mul1=True it is passed as the
+    mul1 marker instead (exllamav3 LinearEXL3 accepts either)."""
+    cls = load_linear_exl3_cls()
+    marker = mcg.contiguous()
+    return cls(
+        config=None,
+        in_features=int(suh.numel()),
+        out_features=int(svh.numel()),
+        trellis=trellis.contiguous(),
+        suh=suh.contiguous(),
+        svh=svh.contiguous(),
+        mcg=None if mul1 else marker,
+        mul1=marker if mul1 else None,
+        out_dtype=out_dtype,
+        transformers_fix=True,
+    )
+
+
+def execute_exl3_linear(
+    x: torch.Tensor,
+    trellis: torch.Tensor,
+    suh: torch.Tensor,
+    svh: torch.Tensor,
+    mcg: torch.Tensor,
+    *,
+    out_dtype: torch.dtype = torch.float32,
+) -> torch.Tensor:
+    """Real EXL3 expert GEMM entry (LinearEXL3 / exllamav3_ext)."""
+    inner = make_linear_exl3(trellis, suh, svh, mcg, out_dtype=torch.float16)
+    return inner.forward(x.contiguous().half(), {}, out_dtype=out_dtype)
+
+
+def fused_moe_enabled() -> bool:
+    return os.environ.get("EXL3_FUSED_MOE", "1") != "0"
+
+
+def load_exllamav3_ext():
+    import exllamav3_ext
+
+    return exllamav3_ext
+
+
+def _exl3_moe_accepts_num_active(fn) -> bool:
+    try:
+        import inspect
+
+        if "num_active" in inspect.signature(fn).parameters:
+            return True
+    except (TypeError, ValueError):
+        pass
+    doc = getattr(fn, "__doc__", None) or ""
+    return "num_active" in doc or "arg29" in doc or doc.count("arg") >= 30
+
+
+def pin_exl3_expert_map(
+    layer: torch.nn.Module, device: torch.device
+) -> torch.Tensor | None:
+    """Move expert_map onto `device` once. CUDA graph capture forbids a CPU→GPU copy."""
+    emap = getattr(layer, "expert_map", None)
+    if emap is None:
+        return None
+    if emap.device != device or emap.dtype != torch.long:
+        layer.expert_map = emap.to(device=device, dtype=torch.long)
+    return layer.expert_map
+
+
+def map_topk_to_local(
+    ids: torch.Tensor,
+    n_local: int,
+    expert_map: torch.Tensor | None,
+) -> torch.Tensor:
+    """ids (T, K) global expert ids → local ids, invalid/non-local → n_local sentinel.
+
+    `expert_map` must already live on `ids.device` (see pin_exl3_expert_map).
+    """
+    flat = ids.reshape(-1)
+    if expert_map is None:
+        invalid = (flat < 0) | (flat >= n_local)
+        return torch.where(invalid, flat.new_full(flat.shape, n_local), flat)
+    if expert_map.device != flat.device or expert_map.dtype != torch.long:
+        raise RuntimeError(
+            "EXL3 expert_map is not pinned to the hidden-state device; "
+            "call pin_exl3_expert_map before fused apply (CUDA graphs forbid the copy)"
+        )
+    n_global = int(expert_map.numel())
+    safe = flat.clamp(min=0, max=max(n_global - 1, 0))
+    mapped = expert_map[safe] if n_global else flat.new_full(flat.shape, n_local)
+    invalid = (flat < 0) | (flat >= n_global) | (mapped < 0) | (mapped >= n_local)
+    return torch.where(invalid, flat.new_full(flat.shape, n_local), mapped)
+
+
+def apply_exl3_python_loop(
+    x2d: torch.Tensor,
+    ids: torch.Tensor,
+    weights: torch.Tensor,
+    inners: list[dict[str, Any]],
+    expert_map: torch.Tensor | None,
+    limit: float,
+    *,
+    only_experts: set[int] | None = None,
+    out: torch.Tensor | None = None,
+) -> torch.Tensor:
+    """Unique-expert LinearEXL3 loop. `only_experts` is local ids (fat-expert fallback)."""
+    tokens, hidden = x2d.shape
+    if out is None:
+        out = torch.zeros(tokens, hidden, dtype=torch.float32, device=x2d.device)
+    unique = torch.unique(ids)
+    for raw in unique.tolist():
+        e_raw = int(raw)
+        if e_raw < 0:
+            continue
+        e = e_raw
+        if expert_map is not None:
+            mapped = int(expert_map[e].item()) if expert_map.numel() > e else e
+            if mapped < 0:
+                continue
+            e = mapped
+        if e >= len(inners):
+            continue
+        if only_experts is not None and e not in only_experts:
+            continue
+        token_idx, k_pos = (ids == int(raw)).nonzero(as_tuple=True)
+        h = x2d.index_select(0, token_idx)
+        pack = inners[e]
+        gate = pack["gate"].forward(h.contiguous().half(), {}, out_dtype=torch.float32)
+        up = pack["up"].forward(h.contiguous().half(), {}, out_dtype=torch.float32)
+        act = F.silu(gate.clamp(max=limit)) * up.clamp(min=-limit, max=limit)
+        down = pack["down"].forward(act.contiguous().half(), {}, out_dtype=torch.float32)
+        scale = weights[token_idx, k_pos].unsqueeze(-1).to(dtype=torch.float32)
+        out.index_add_(0, token_idx, down * scale)
+    return out
+
+
+def build_exl3_fused_state(layer: torch.nn.Module, inners: list[dict[str, Any]]) -> None:
+    """Pointer tables + fused temps, once after load. No per-token alloc."""
+    import exllamav3_ext
+
+    device = layer.w13_trellis.device
+    n_exp = len(inners)
+    hidden = int(layer._exl3_hidden_size)
+    intermediate = int(layer._exl3_intermediate_local)
+
+    def _ptrs(which: str, attr: str) -> torch.Tensor:
+        return torch.tensor(
+            [int(getattr(pack[which], attr).data_ptr()) for pack in inners],
+            dtype=torch.int64,
+            device=device,
+        )
+
+    layer._exl3_ptrs = {
+        "gate_trellis": _ptrs("gate", "trellis"),
+        "gate_suh": _ptrs("gate", "suh"),
+        "gate_svh": _ptrs("gate", "svh"),
+        "up_trellis": _ptrs("up", "trellis"),
+        "up_suh": _ptrs("up", "suh"),
+        "up_svh": _ptrs("up", "svh"),
+        "down_trellis": _ptrs("down", "trellis"),
+        "down_suh": _ptrs("down", "suh"),
+        "down_svh": _ptrs("down", "svh"),
+    }
+    idx = int(device.index) if device.index is not None else 0
+    concurrency = int(exllamav3_ext.exl3_moe_max_concurrency(idx))
+    if concurrency < 1:
+        concurrency = 1
+    key = (str(device), hidden, intermediate, concurrency)
+    temps = _FUSED_TEMP_CACHE.get(key)
+    if temps is None:
+        temps = (
+            torch.empty((concurrency, TEMP_ROWS_FUSED, hidden), dtype=torch.float16, device=device),
+            torch.empty((concurrency, TEMP_ROWS_FUSED, hidden), dtype=torch.float16, device=device),
+            torch.empty((concurrency, TEMP_ROWS_FUSED, intermediate), dtype=torch.float16, device=device),
+            torch.empty((concurrency, TEMP_ROWS_FUSED, intermediate), dtype=torch.float16, device=device),
+        )
+        _FUSED_TEMP_CACHE[key] = temps
+    layer._exl3_fused_temps = temps
+    layer._exl3_fused_concurrency = concurrency
+    layer._exl3_k = int(layer._exl3_bits)
+
+
+def apply_exl3_fused_moe(
+    x2d: torch.Tensor,
+    ids: torch.Tensor,
+    weights: torch.Tensor,
+    layer: torch.nn.Module,
+    inners: list[dict[str, Any]],
+    expert_map: torch.Tensor | None,
+    limit: float,
+) -> torch.Tensor:
+    """One exl3_moe launch per layer. Experts with count > 128 fall back to LinearEXL3."""
+    import exllamav3_ext
+
+    tokens, hidden = x2d.shape
+    n_exp = len(inners)
+    ptrs = getattr(layer, "_exl3_ptrs", None)
+    temps = getattr(layer, "_exl3_fused_temps", None)
+    if not ptrs or temps is None:
+        raise RuntimeError("EXL3 fused pointer tables were not built after weight load")
+
+    local = map_topk_to_local(ids, n_exp, expert_map)
+    topk = int(ids.shape[-1])
+    flat_token = torch.arange(tokens, device=x2d.device, dtype=torch.long).repeat_interleave(topk)
+    flat_weight = weights.reshape(-1).to(dtype=torch.float16)
+    order = local.argsort()
+    token_sorted = flat_token[order]
+    weight_sorted = flat_weight[order]
+    # scatter_add stays on GPU. torch.bincount can host-stage and break CUDA graphs.
+    expert_count = torch.zeros(n_exp + 1, dtype=torch.long, device=local.device)
+    expert_count.scatter_add_(
+        0, local.long(), torch.ones(local.shape, dtype=torch.long, device=local.device)
+    )
+    out = torch.zeros(tokens, hidden, dtype=torch.float32, device=x2d.device)
+    xh = x2d.contiguous().half()
+
+    counts = expert_count[:n_exp]
+    fn = exllamav3_ext.exl3_moe
+    # -1 = unknown active count: max-concurrency grid, no .item() host sync.
+    n_active_host = -1 if _exl3_moe_accepts_num_active(fn) else None
+
+    k = int(getattr(layer, "_exl3_k", 4))
+    _m1 = bool(getattr(layer, "_exl3_mul1", False))
+    args = (
+        xh,
+        out,
+        expert_count,
+        token_sorted,
+        weight_sorted,
+        temps[0],
+        temps[1],
+        temps[2],
+        temps[3],
+        MOE_ACT_SILU,
+        k,
+        k,
+        k,
+        ptrs["gate_trellis"],
+        ptrs["gate_suh"],
+        ptrs["gate_svh"],
+        ptrs["up_trellis"],
+        ptrs["up_suh"],
+        ptrs["up_svh"],
+        ptrs["down_trellis"],
+        ptrs["down_suh"],
+        ptrs["down_svh"],
+        # KEYS PATCH: (mcg, mul1) flag pairs for gate / up / down, per-layer codebook.
+        not _m1,
+        _m1,
+        not _m1,
+        _m1,
+        not _m1,
+        _m1,
+        float(limit),
+    )
+    if n_active_host is not None:
+        fn(*args, n_active_host)
+    else:
+        fn(*args)
+
+    if tokens > TEMP_ROWS_FUSED:
+        fat = (counts > TEMP_ROWS_FUSED).nonzero(as_tuple=False).view(-1)
+        if fat.numel():
+            apply_exl3_python_loop(
+                x2d,
+                ids,
+                weights,
+                inners,
+                expert_map,
+                limit,
+                only_experts=set(int(i) for i in fat.tolist()),
+                out=out,
+            )
+    return out
+
+
+def apply_exl3_experts(
+    x: torch.Tensor,
+    topk_ids: torch.Tensor,
+    topk_weights: torch.Tensor,
+    layer: torch.nn.Module,
+    *,
+    limit: float = SWIGLU_LIMIT_DEFAULT,
+    fused: bool | None = None,
+) -> torch.Tensor:
+    """Shipped routed-expert apply. `fused=None` honors EXL3_FUSED_MOE."""
+    inners = getattr(layer, "_exl3_inners", None)
+    if not inners:
+        raise RuntimeError("EXL3 experts were not built after weight load")
+    tokens, hidden = x.shape[-2], x.shape[-1]
+    x2d = x.reshape(tokens, hidden)
+    ids = topk_ids.reshape(tokens, -1).to(torch.long)
+    weights = topk_weights.reshape(tokens, -1)
+    expert_map = pin_exl3_expert_map(layer, x2d.device)
+    have_ptrs = bool(getattr(layer, "_exl3_ptrs", None))
+    if fused is True and not have_ptrs:
+        raise RuntimeError("EXL3 fused apply requested but pointer tables are missing")
+    use_fused = (fused_moe_enabled() if fused is None else bool(fused)) and have_ptrs
+    if use_fused:
+        try:
+            import exllamav3_ext
+
+            use_fused = hasattr(exllamav3_ext, "exl3_moe")
+        except Exception:
+            use_fused = False
+    if use_fused:
+        out = apply_exl3_fused_moe(x2d, ids, weights, layer, inners, expert_map, limit)
+        layer._exl3_last_apply = "fused"
+    else:
+        out = apply_exl3_python_loop(x2d, ids, weights, inners, expert_map, limit)
+        layer._exl3_last_apply = "loop"
+    return out.to(dtype=x.dtype)
+
+
+def _suffix_from_mapped_name(weight_name: str) -> str:
+    tail = weight_name.rsplit(".", 1)[-1]
+    for suffix in EXL3_SUFFIXES:
+        if tail == suffix or tail.endswith("_" + suffix):
+            return suffix
+    raise ValueError(f"not an EXL3 packed name: {weight_name}")
+
+
+
+def _is_mtp_prefix(prefix: str) -> bool:
+    """True for layers at index >= num_hidden_layers (DeepSeek/GLM MTP blocks)."""
+    import re as _re
+    m = _re.search(r"\.layers\.(\d+)\.", prefix + ".")
+    if not m:
+        return "mtp" in prefix.lower()
+    idx = int(m.group(1))
+    n = None
+    try:
+        from vllm.config import get_current_vllm_config
+        n = int(get_current_vllm_config().model_config.hf_config.num_hidden_layers)
+    except Exception:  # noqa: BLE001
+        n = None
+    if n is None:
+        n = int(os.environ.get("EXL3_MTP_LAYER_START", "78"))
+    return idx >= n
+
+@register_quantization_config("exl3")
+class Exl3Config(QuantizationConfig):
+    """Routed-experts-only EXL3/MCG. Dense / shared / attention stay native."""
+
+    def __init__(
+        self,
+        bits: int = 4,
+        codebook: str = "mcg",
+        scope: str = "glm53_routed_experts_only",
+        **kwargs: Any,
+    ) -> None:
+        super().__init__()
+        self.bits = int(bits)
+        self.codebook = str(codebook)
+        self.scope = str(scope)
+        self.raw_config = dict(kwargs)
+        if self.codebook not in ("mcg", "mul1", "mixed", "auto"):
+            raise ValueError(
+                f"this overlay implements codebook mcg/mul1 (per layer); got {self.codebook!r}"
+            )
+        if self.bits not in (3, 4, 5, 6):
+            raise ValueError(f"unsupported EXL3 bits={self.bits}")
+
+    def get_name(self) -> str:
+        return "exl3"
+
+    def get_supported_act_dtypes(self) -> list[torch.dtype]:
+        return [torch.bfloat16, torch.float16, torch.float32]
+
+    @classmethod
+    def get_min_capability(cls) -> int:
+        # LinearEXL3 uses CUDA >= Ampere; GB10 is SM121.
+        return 80
+
+    @staticmethod
+    def get_config_filenames() -> list[str]:
+        return ["quantization_config.json"]
+
+    @classmethod
+    def from_config(cls, config: dict[str, Any]) -> "Exl3Config":
+        skip = {
+            "bits",
+            "codebook",
+            "scope",
+            "quant_method",
+            # tr3 ships a 37 MiB per-tensor ledger; keep it off the config object.
+            "tensor_storage",
+        }
+        return cls(
+            bits=int(config.get("bits", 4)),
+            codebook=str(config.get("codebook", "mcg")),
+            scope=str(config.get("scope", "glm53_routed_experts_only")),
+            **{k: v for k, v in config.items() if k not in skip},
+        )
+
+    @classmethod
+    def override_quantization_method(
+        cls,
+        hf_quant_cfg: dict[str, Any],
+        user_quant: str | None,
+        hf_config: Any = None,
+    ) -> str | None:
+        method = str((hf_quant_cfg or {}).get("quant_method", "")).lower()
+        if method == "exl3":
+            return "exl3"
+        return None
+
+    def get_quant_method(self, layer: torch.nn.Module, prefix: str):
+        from vllm.model_executor.layers.fused_moe.routed_experts import RoutedExperts
+
+        # KEYS PATCH: the MTP block (model.layers.<num_hidden_layers>+) is stored bf16
+        # (-mb 16) -> leave it to the stock unquantized MoE path so SPEC_METHOD=mtp works.
+        if isinstance(layer, RoutedExperts) and _is_mtp_prefix(prefix):
+            logger.info_once("EXL3: MTP block %s left unquantized (bf16 experts)", prefix)
+            return None
+        if isinstance(layer, RoutedExperts):
+            return Exl3MoEMethod(layer.moe_config, self)
+        if isinstance(layer, LinearBase):
+            return UnquantizedLinearMethod()
+        return None
+
+
+class Exl3MoEMethod(FusedMoEMethodBase):
+    """Packed MCG trellis experts: create/load packed tensors, LinearEXL3 apply."""
+
+    def __init__(self, moe, quant_config: Exl3Config) -> None:
+        super().__init__(moe)
+        self.quant_config = quant_config
+        self.bits = quant_config.bits
+        self._logged = False
+
+    def get_fused_moe_quant_config(self, layer: "RoutedExperts") -> FusedMoEQuantConfig | None:
+        return None
+
+    def create_weights(
+        self,
+        layer: "RoutedExperts",
+        num_experts: int,
+        hidden_size: int,
+        intermediate_size_per_partition: int,
+        params_dtype: torch.dtype,
+        **extra_weight_attrs,
+    ) -> None:
+        del params_dtype
+        if hidden_size % 16 or intermediate_size_per_partition % 16:
+            raise ValueError(
+                "EXL3 trellis tiles are 16-wide; "
+                f"hidden={hidden_size} intermediate_local={intermediate_size_per_partition}"
+            )
+        k_words = self.bits * 16
+        in_tiles = hidden_size // 16
+        out_tiles = intermediate_size_per_partition // 16
+
+        extra = {k: v for k, v in extra_weight_attrs.items() if k != "weight_loader"}
+
+        # w13_* : stacked [expert, {gate=0, up=1}, ...] so the stock
+        # expert_params_mapping (experts.w13_ + suffix) hits these names.
+        w13_trellis = Parameter(
+            torch.empty(
+                num_experts, 2, in_tiles, out_tiles, k_words, dtype=torch.int16
+            ),
+            requires_grad=False,
+        )
+        w13_suh = Parameter(
+            torch.empty(num_experts, 2, hidden_size, dtype=torch.float16),
+            requires_grad=False,
+        )
+        w13_svh = Parameter(
+            torch.empty(
+                num_experts, 2, intermediate_size_per_partition, dtype=torch.float16
+            ),
+            requires_grad=False,
+        )
+        w13_mcg = Parameter(
+            torch.zeros(num_experts, 2, 1, dtype=torch.int32),
+            requires_grad=False,
+        )
+        w13_mul1 = Parameter(
+            torch.zeros(num_experts, 2, 1, dtype=torch.int32),
+            requires_grad=False,
+        )
+        w2_trellis = Parameter(
+            torch.empty(
+                num_experts, out_tiles, in_tiles, k_words, dtype=torch.int16
+            ),
+            requires_grad=False,
+        )
+        w2_suh = Parameter(
+            torch.empty(
+                num_experts, intermediate_size_per_partition, dtype=torch.float16
+            ),
+            requires_grad=False,
+        )
+        w2_svh = Parameter(
+            torch.empty(num_experts, hidden_size, dtype=torch.float16),
+            requires_grad=False,
+        )
+        w2_mcg = Parameter(
+            torch.zeros(num_experts, 1, dtype=torch.int32),
+            requires_grad=False,
+        )
+        w2_mul1 = Parameter(
+            torch.zeros(num_experts, 1, dtype=torch.int32),
+            requires_grad=False,
+        )
+
+        packed = {
+            "w13_trellis": w13_trellis,
+            "w13_suh": w13_suh,
+            "w13_svh": w13_svh,
+            "w13_mcg": w13_mcg,
+            "w13_mul1": w13_mul1,
+            "w2_trellis": w2_trellis,
+            "w2_suh": w2_suh,
+            "w2_svh": w2_svh,
+            "w2_mcg": w2_mcg,
+            "w2_mul1": w2_mul1,
+        }
+        for name, param in packed.items():
+            layer.register_parameter(name, param)
+            set_weight_attrs(param, extra)
+            param.weight_loader = self._load_exl3
+            param._exl3_owner = layer
+        if hasattr(layer, "w13_weight") or hasattr(layer, "w2_weight"):
+            raise RuntimeError("EXL3 create_weights must not allocate dense expert weights")
+
+        layer._exl3_hidden_size = hidden_size
+        layer._exl3_intermediate_local = intermediate_size_per_partition
+        layer._exl3_k_words = k_words
+        layer._exl3_bits = self.bits
+
+    def _load_exl3(
+        self,
+        param: Parameter,
+        loaded_weight: torch.Tensor,
+        weight_name: str,
+        shard_id: str = "w1",
+        expert_id: int = 0,
+        return_success: bool = False,
+    ) -> bool | None:
+        from vllm.distributed import (
+            get_tensor_model_parallel_rank,
+            get_tensor_model_parallel_world_size,
+        )
+
+        layer = param
+        # param is the Parameter; expert_id is already physical. Map to local
+        # via the owning module if present on the weight_loader closure... we
+        # look up from param's __dict__ after register. RoutedExperts.weight_loader
+        # maps global→local; glm5next calls *our* loader, so map here.
+        owner = getattr(param, "_exl3_owner", None)
+        if owner is not None:
+            local_id = owner._map_global_expert_id_to_local_expert_id(expert_id)
+            if local_id == -1:
+                return False if return_success else None
+            expert_id = local_id
+
+        tp_rank = get_tensor_model_parallel_rank()
+        tp_size = get_tensor_model_parallel_world_size()
+        suffix = _suffix_from_mapped_name(weight_name)
+        loaded = loaded_weight.detach().contiguous()
+
+        if shard_id in ("w1", "w3"):
+            shard_idx = 0 if shard_id == "w1" else 1
+            sharded = shard_exl3_col(loaded, suffix, tp_rank, tp_size)
+            dest = param.data[expert_id, shard_idx]
+        elif shard_id == "w2":
+            sharded = shard_exl3_row(loaded, suffix, tp_rank, tp_size)
+            dest = param.data[expert_id]
+        else:
+            raise ValueError(f"unknown EXL3 shard_id={shard_id}")
+
+        # KEYS PATCH: codebook markers (mcg/mul1) may be stored as 0-d scalars; accept any
+        # shape with the same element count.
+        if suffix in ("mcg", "mul1") and sharded.numel() == dest.numel():
+            sharded = sharded.reshape(dest.shape)
+        if tuple(dest.shape) != tuple(sharded.shape):
+            raise RuntimeError(
+                f"EXL3 load shape mismatch {weight_name} shard={shard_id} "
+                f"expert={expert_id}: dest {tuple(dest.shape)} != "
+                f"loaded {tuple(sharded.shape)}"
+            )
+        dest.copy_(sharded)
+        return True if return_success else None
+
+    def process_weights_after_loading(self, layer: torch.nn.Module) -> None:
+        if not hasattr(layer, "w13_trellis"):
+            return
+        # Bind owner for any late loads; stitch LinearEXL3 handles.
+        for name in (
+            "w13_trellis",
+            "w13_suh",
+            "w13_svh",
+            "w13_mcg",
+            "w13_mul1",
+            "w2_trellis",
+            "w2_suh",
+            "w2_svh",
+            "w2_mcg",
+            "w2_mul1",
+        ):
+            getattr(layer, name)._exl3_owner = layer
+
+        # KEYS PATCH: per-layer codebook detection from the loaded marker tensors.
+        mcg13 = layer.w13_mcg.reshape(-1)
+        mcg2 = layer.w2_mcg.reshape(-1)
+        mul13 = layer.w13_mul1.reshape(-1)
+        mul2 = layer.w2_mul1.reshape(-1)
+        is_mcg = bool(torch.all(mcg13 == MCG_MARKER_SIGNED_INT32)) and bool(
+            torch.all(mcg2 == MCG_MARKER_SIGNED_INT32)
+        )
+        is_mul1 = bool(torch.all(mul13 == MUL1_MARKER_SIGNED_INT32)) and bool(
+            torch.all(mul2 == MUL1_MARKER_SIGNED_INT32)
+        )
+        if is_mcg == is_mul1:
+            raise RuntimeError(
+                "EXL3 codebook markers inconsistent for this layer: "
+                f"mcg={is_mcg} mul1={is_mul1} (expect exactly one; "
+                f"MCG {MCG_MARKER_SIGNED_INT32} / MUL1 {MUL1_MARKER_SIGNED_INT32})"
+            )
+        use_mul1 = is_mul1
+        layer._exl3_mul1 = use_mul1
+        m13 = layer.w13_mul1 if use_mul1 else layer.w13_mcg
+        m2 = layer.w2_mul1 if use_mul1 else layer.w2_mcg
+        if not getattr(Exl3MoEMethod, "_logged_codebooks", None):
+            Exl3MoEMethod._logged_codebooks = set()
+        if use_mul1 not in Exl3MoEMethod._logged_codebooks:
+            Exl3MoEMethod._logged_codebooks.add(use_mul1)
+            logger.info("EXL3 experts codebook detected: %s", "mul1" if use_mul1 else "mcg")
+
+        n_exp = int(layer.w13_trellis.shape[0])
+        inners: list[dict[str, Any]] = []
+        for e in range(n_exp):
+            gate = make_linear_exl3(
+                layer.w13_trellis[e, 0],
+                layer.w13_suh[e, 0],
+                layer.w13_svh[e, 0],
+                m13[e, 0],
+                mul1=use_mul1,
+            )
+            up = make_linear_exl3(
+                layer.w13_trellis[e, 1],
+                layer.w13_suh[e, 1],
+                layer.w13_svh[e, 1],
+                m13[e, 1],
+                mul1=use_mul1,
+            )
+            down = make_linear_exl3(
+                layer.w2_trellis[e],
+                layer.w2_suh[e],
+                layer.w2_svh[e],
+                m2[e],
+                mul1=use_mul1,
+            )
+            inners.append({"gate": gate, "up": up, "down": down})
+        layer._exl3_inners = inners
+        fused_ok = False
+        fused_err = None
+        if fused_moe_enabled():
+            try:
+                import exllamav3_ext
+
+                if hasattr(exllamav3_ext, "exl3_moe"):
+                    build_exl3_fused_state(layer, inners)
+                    fused_ok = True
+                else:
+                    fused_err = "exllamav3_ext.exl3_moe missing"
+            except Exception as exc:
+                fused_err = repr(exc)
+                layer._exl3_ptrs = None
+        if not self._logged:
+            if fused_ok:
+                logger.info(
+                    "EXL3 MCG trellis engaged for routed experts: bits=%s "
+                    "experts_local=%s hidden=%s intermediate_local=%s "
+                    "fused_moe=exl3_moe concurrency=%s "
+                    "(no BF16 expert reconstruct at load)",
+                    self.bits,
+                    n_exp,
+                    layer._exl3_hidden_size,
+                    layer._exl3_intermediate_local,
+                    getattr(layer, "_exl3_fused_concurrency", "?"),
+                )
+            else:
+                logger.info(
+                    "EXL3 MCG trellis engaged for routed experts: bits=%s "
+                    "experts_local=%s hidden=%s intermediate_local=%s "
+                    "fused_moe=python_loop (%s) "
+                    "(no BF16 expert reconstruct at load)",
+                    self.bits,
+                    n_exp,
+                    layer._exl3_hidden_size,
+                    layer._exl3_intermediate_local,
+                    fused_err or "EXL3_FUSED_MOE=0",
+                )
+            self._logged = True
+
+    def apply(
+        self,
+        layer: "RoutedExperts",
+        x: torch.Tensor,
+        topk_weights: torch.Tensor,
+        topk_ids: torch.Tensor,
+        shared_experts: "SharedExperts | None",
+        shared_experts_input: torch.Tensor | None,
+    ) -> torch.Tensor:
+        del shared_experts, shared_experts_input
+        limit = getattr(self.moe, "swiglu_limit", None) or SWIGLU_LIMIT_DEFAULT
+        return apply_exl3_experts(
+            x, topk_ids, topk_weights, layer, limit=float(limit)
+        )

serving/finalize_config.py

@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""Post-compile fixups so the exllamav3 output is a Mia-native vLLM checkpoint.
+
+* config.json / quantization_config.json: quant_method=exl3, bits=<int>, codebook=mcg,
+  scope=glm53_routed_experts_only, head_bits=16, mtp_bits=16 (Mia's Exl3Config reads bits/codebook/scope;
+  everything else is carried as raw config). The FP8 source block is preserved as original_quantization_config.
+* sanity: every routed expert has trellis/suh/svh/mcg; every non-expert linear is a plain .weight; report dtypes.
+Usage: finalize_config.py <out_dir> [expert_bits]
+"""
+import json, os, sys, re, struct, collections
+
+out = sys.argv[1]
+bits = int(sys.argv[2]) if len(sys.argv) > 2 else 3
+CODEBOOK = sys.argv[3] if len(sys.argv) > 3 else "mcg"
+
+def read_header(path):
+    with open(path, "rb") as f:
+        n = struct.unpack("<Q", f.read(8))[0]
+        return json.loads(f.read(n))
+
+# --- tensor inventory
+suffix_by_kind = collections.Counter()
+dtypes = collections.Counter()
+experts_ok, experts_bad = 0, []
+nonexp_quant = []
+shards = sorted(p for p in os.listdir(out) if p.endswith(".safetensors"))
+seen = {}
+for sh in shards:
+    for k, v in read_header(os.path.join(out, sh)).items():
+        if k == "__metadata__":
+            continue
+        seen[k] = v["dtype"]
+exp = collections.defaultdict(set)
+for k in seen:
+    m = re.match(r"model\.layers\.(\d+)\.mlp\.experts\.(\d+)\.(gate|up|down)_proj\.(\w+)$", k)
+    if m:
+        exp[(int(m.group(1)), int(m.group(2)), m.group(3))].add(m.group(4))
+        continue
+    m2 = re.match(r"(model\.layers\.\d+\.(self_attn|mlp\.shared_experts|mlp)\.[\w.]+?)\.(trellis|suh|svh|mcg|mul1)$", k)
+    if m2:
+        nonexp_quant.append(k)
+for key, sfx in exp.items():
+    if {"trellis", "suh", "svh"} <= sfx and ({"mcg"} <= sfx or {"mul1"} <= sfx):
+        experts_ok += 1
+    else:
+        experts_bad.append((key, sorted(sfx)))
+for k, d in seen.items():
+    dtypes[(k.rsplit(".", 1)[-1], d)] += 1
+print(f"shards={len(shards)} tensors={len(seen)} experts_ok={experts_ok} experts_bad={len(experts_bad)} nonexpert_quantized={len(nonexp_quant)}")
+if experts_bad:
+    print("  bad:", experts_bad[:5])
+if nonexp_quant:
+    print("  !! non-expert EXL3 tensors (Mia overlay expects native):", nonexp_quant[:5])
+for (sfx, d), n in sorted(dtypes.items(), key=lambda x: -x[1])[:12]:
+    print(f"  {sfx:14s} {d:8s} {n}")
+
+# --- config fixups
+def fix(path, is_hf_config):
+    d = json.load(open(path))
+    q = d.get("quantization_config", d if not is_hf_config else {}) if is_hf_config else d
+    if is_hf_config:
+        q = d.setdefault("quantization_config", {})
+    orig = None
+    if q.get("quant_method") not in (None, "exl3"):
+        orig = dict(q)
+        q.clear()
+    if orig is not None:
+        q["original_quantization_config"] = orig
+    q.update({"quant_method": "exl3", "bits": bits, "codebook": CODEBOOK, "scope": "glm53_routed_experts_only",
+              "head_bits": 16, "mtp_bits": 16, "keys_profile": f"routed-experts-{bits}bpw-{CODEBOOK}; attn/dense/shared/head bf16",
+              "keys_source": "zai-org/GLM-5.3 (FP8) via exllamav3 1.4.5 + keys encode farm"})
+    q.pop("tensor_storage", None) if is_hf_config else None
+    json.dump(d, open(path, "w"), indent=2)
+    print(f"wrote {path}: quant_method={q['quant_method']} bits={q['bits']} codebook={q['codebook']}")
+
+fix(os.path.join(out, "config.json"), True)
+qc = os.path.join(out, "quantization_config.json")
+if os.path.exists(qc):
+    fix(qc, False)
+for fn in ("tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "generation_config.json", "model.safetensors.index.json"):
+    print(f"  {fn}: {'ok' if os.path.exists(os.path.join(out, fn)) else 'MISSING'}")

serving/glm53_probe.py

@@ -0,0 +1,32 @@
+import json,time,urllib.request,sys
+BASE="http://10.100.10.1:8888"
+def chat(prompt,max_tokens,think=False):
+    body={"model":"GLM-5.3-EXL3","messages":[{"role":"user","content":prompt}],"max_tokens":max_tokens,"temperature":0,"stream":True,"stream_options":{"include_usage":True},"chat_template_kwargs":{"enable_thinking":think}}
+    req=urllib.request.Request(BASE+"/v1/chat/completions",data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
+    t0=time.time(); ttft=None; n=0; text=""
+    with urllib.request.urlopen(req,timeout=3600) as r:
+        for line in r:
+            line=line.decode().strip()
+            if not line.startswith("data:") or line=="data: [DONE]": continue
+            ev=json.loads(line[5:])
+            if ev.get("usage"): n=ev["usage"]["completion_tokens"]; pt=ev["usage"]["prompt_tokens"]
+            ch=ev.get("choices")
+            if ch:
+                d=ch[0].get("delta") or {}
+                c=d.get("content") or d.get("reasoning_content") or ""
+                if c and ttft is None: ttft=time.time()-t0
+                text+=c
+    wall=time.time()-t0
+    return dict(prompt_tokens=pt,tokens=n,ttft=ttft,wall=wall,decode_tps=n/(wall-ttft) if ttft and wall>ttft else None,text=text)
+mode=sys.argv[1]
+if mode=="decode":
+    r=chat("Write a detailed 300-word explanation of how a transformer language model generates text, covering tokenization, attention, and sampling.",256)
+    print(f"decode: prompt {r['prompt_tokens']} tok, gen {r['tokens']} tok, TTFT {r['ttft']:.2f}s, decode {r['decode_tps']:.2f} tok/s"); print("head:",repr(r["text"][:200]))
+elif mode=="needle":
+    k=int(sys.argv[2])
+    filler="The quick brown fox jumps over the lazy dog. Pack my box with five dozen liquor jugs. "*(k*1000//19)
+    secret="The secret passphrase is VIOLET-KANGAROO-7731."
+    pos=len(filler)//2; doc=filler[:pos]+" "+secret+" "+filler[pos:]
+    r=chat(doc+"\n\nWhat is the secret passphrase mentioned in the text above? Reply with just the passphrase.",40)
+    ok="VIOLET-KANGAROO-7731" in r["text"]
+    print(f"needle {k}K: prompt {r['prompt_tokens']} tok, TTFT {r['ttft']:.1f}s ({r['prompt_tokens']/r['ttft']:.0f} tok/s prefill), decode {r['decode_tps']:.2f} tok/s, PASS={ok}, reply={r['text'].strip()[:80]!r}")

serving/keys-launch.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+# keys wrapper: drop page caches on all 4 ranks (NFS weight reads inflate "used" and trip vLLM's
+# startup free-memory guard on unified-memory GB10), then launch with the standard skip flags.
+# Usage: [ENFORCE_EAGER=0] [EXL3_FUSED_MOE=1] [GPU_MEM_UTIL=0.86] ./keys-launch.sh [logname]
+set -u
+cd ~/glm53-mia
+LOG=${1:-launch-tp4-keys.log}
+./start-tp4.sh stop 2>&1 | tail -1
+for ip in 10.100.10.1 10.100.10.2 10.100.10.3 10.100.10.5; do
+  ssh -o BatchMode=yes -o ConnectTimeout=10 "$ip" "sudo sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null; free -g | awk '/Mem:/{print \"\" \" free=\" \$4 \"G avail=\" \$7 \"G\"}'" 2>/dev/null || echo "$ip: drop_caches failed"
+done
+SKIP_PULL=1 SKIP_SHIP=1 SKIP_BUILD=1 SKIP_DOWNLOAD=1 SKIP_SYNC=1 SKIP_OVERLAY_VERIFY=1 \
+  ENFORCE_EAGER=${ENFORCE_EAGER:-0} EXL3_FUSED_MOE=${EXL3_FUSED_MOE:-1} \
+  setsid nohup ./start-tp4.sh > "$LOG" 2>&1 < /dev/null &
+echo "launched -> $LOG"

serving/patch_ablit_dsv2.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+"""Install Mia's ABLIT load-time hook for FULL GLM-5.3 (GlmMoeDsaForCausalLM -> deepseek_v2.py).
+
+Mia's overlay/patch_ablit.py splices the hook into the GLM-5.3-Flash classes (glm5next). The full
+model is served by DeepseekV2Model / DeepSeekMTP, so this splices the same call at the end of their
+load_weights. ablit_runtime.py is reused unchanged: its o_proj regexes already match
+``model.layers.N.self_attn.o_proj`` and ``layers.N.mtp_block.self_attn.o_proj``. Idempotent.
+Donor set: <ABLIT_DIR>/transplant/L{N}.bin + MANIFEST.json (make_derisk_donor.py output).
+Runtime env: ABLIT=1 ABLIT_METHOD=transplant ABLIT_LAYERS=<ranges> ABLIT_INCLUDE_MTP=1 ABLIT_DIR=/opt/glm53/ablit
+"""
+from pathlib import Path
+import shutil, sys
+
+SITE = Path("/usr/local/lib/python3.12/dist-packages/vllm")
+MODELS = SITE / "model_executor/models"
+NVIDIA_DIR = SITE / "models/glm5next/nvidia"
+RUNTIME_SRC = Path(sys.argv[1] if len(sys.argv) > 1 else "/opt/glm53/ablit_runtime.py")
+RUNTIME_DST = NVIDIA_DIR / "glm53_ablit.py"
+MARKER = "ABLIT-HOOK-DSV2"
+HOOK = (
+    "        from vllm.models.glm5next.nvidia.glm53_ablit import maybe_apply "
+    "as _glm53_ablit_maybe_apply  # ABLIT-HOOK-DSV2\n"
+    "        _glm53_ablit_maybe_apply(self)  # ABLIT-HOOK-DSV2\n"
+)
+MODEL_TAIL = """            if name is not None and not is_fusion_moe_shared_experts_layer:
+                loaded_params.add(name)
+
+        return loaded_params"""
+MTP_TAIL = """                    f"or disable speculative decoding."
+                )
+
+        return loaded_params"""
+
+
+def splice(path: Path, tail: str, label: str) -> None:
+    text = path.read_text()
+    if MARKER in text:
+        print(f"{path.name}: {MARKER} already present")
+        return
+    n = text.count(tail)
+    if n != 1:
+        raise SystemExit(f"{path}: expected one patch target for {label}, found {n}")
+    new_tail = tail.replace("        return loaded_params", HOOK + "        return loaded_params")
+    path.write_text(text.replace(tail, new_tail))
+    print(f"patched {path.name} ({label})")
+
+
+NVIDIA_DIR.mkdir(parents=True, exist_ok=True)
+if RUNTIME_SRC.is_file():
+    shutil.copyfile(RUNTIME_SRC, RUNTIME_DST)
+    print(f"installed {RUNTIME_DST}")
+elif not RUNTIME_DST.is_file():
+    raise SystemExit(f"ablit runtime not found at {RUNTIME_SRC}")
+splice(MODELS / "deepseek_v2.py", MODEL_TAIL, "DeepseekV2Model.load_weights")
+splice(MODELS / "deepseek_mtp.py", MTP_TAIL, "DeepSeekMTP.load_weights")

serving/patch_overlay_mtp.py

@@ -0,0 +1,40 @@
+import sys
+p=sys.argv[1]; s=open(p).read()
+old="""    def get_quant_method(self, layer: torch.nn.Module, prefix: str):
+        from vllm.model_executor.layers.fused_moe.routed_experts import RoutedExperts
+
+        if isinstance(layer, RoutedExperts):
+            return Exl3MoEMethod(layer.moe_config, self)"""
+new="""    def get_quant_method(self, layer: torch.nn.Module, prefix: str):
+        from vllm.model_executor.layers.fused_moe.routed_experts import RoutedExperts
+
+        # KEYS PATCH: the MTP block (model.layers.<num_hidden_layers>+) is stored bf16
+        # (-mb 16) -> leave it to the stock unquantized MoE path so SPEC_METHOD=mtp works.
+        if isinstance(layer, RoutedExperts) and _is_mtp_prefix(prefix):
+            logger.info_once("EXL3: MTP block %s left unquantized (bf16 experts)", prefix)
+            return None
+        if isinstance(layer, RoutedExperts):
+            return Exl3MoEMethod(layer.moe_config, self)"""
+assert s.count(old)==1, s.count(old); s=s.replace(old,new)
+helper='''
+
+def _is_mtp_prefix(prefix: str) -> bool:
+    """True for layers at index >= num_hidden_layers (DeepSeek/GLM MTP blocks)."""
+    import re as _re
+    m = _re.search(r"\\.layers\\.(\\d+)\\.", prefix + ".")
+    if not m:
+        return "mtp" in prefix.lower()
+    idx = int(m.group(1))
+    n = None
+    try:
+        from vllm.config import get_current_vllm_config
+        n = int(get_current_vllm_config().model_config.hf_config.num_hidden_layers)
+    except Exception:  # noqa: BLE001
+        n = None
+    if n is None:
+        n = int(os.environ.get("EXL3_MTP_LAYER_START", "78"))
+    return idx >= n
+'''
+anchor='\n@register_quantization_config("exl3")'
+assert s.count(anchor)==1, s.count(anchor); s=s.replace(anchor, helper+anchor)
+open(p,"w").write(s); print("mtp exemption patched")

serving/patch_overlay_mul1.py

@@ -0,0 +1,201 @@
+import sys,re
+src=sys.argv[1]; dst=sys.argv[2]; s=open(src).read()
+def rep(old,new,count=1):
+    global s
+    assert old in s, f"missing: {old[:70]!r}"
+    s=s.replace(old,new,count)
+rep('MCG_MARKER_SIGNED_INT32 = -877912083\nEXL3_SUFFIXES = ("trellis", "suh", "svh", "mcg")',
+'''MCG_MARKER_SIGNED_INT32 = -877912083
+# KEYS PATCH: exllamav3 "mul1" codebook (converter default; 0x83DCD12D marker). LinearEXL3 and
+# exllamav3_ext.exl3_moe in this image already take (mcg, mul1) flag pairs per matrix.
+MUL1_MULTIPLIER = 0x83DCD12D
+MUL1_MARKER_SIGNED_INT32 = -2082680531
+EXL3_SUFFIXES = ("trellis", "suh", "svh", "mcg", "mul1")''')
+rep('''    mcg: torch.Tensor,
+    *,
+    out_dtype: torch.dtype = torch.float16,
+):
+    """Build a LinearEXL3 over already-sharded packed tensors. No BF16 expand."""
+    cls = load_linear_exl3_cls()
+    return cls(
+        config=None,
+        in_features=int(suh.numel()),
+        out_features=int(svh.numel()),
+        trellis=trellis.contiguous(),
+        suh=suh.contiguous(),
+        svh=svh.contiguous(),
+        mcg=mcg.contiguous(),
+        out_dtype=out_dtype,
+        transformers_fix=True,
+    )''','''    mcg: torch.Tensor,
+    *,
+    mul1: bool = False,
+    out_dtype: torch.dtype = torch.float16,
+):
+    """Build a LinearEXL3 over already-sharded packed tensors. No BF16 expand.
+    KEYS PATCH: `mcg` is the codebook marker tensor; with mul1=True it is passed as the
+    mul1 marker instead (exllamav3 LinearEXL3 accepts either)."""
+    cls = load_linear_exl3_cls()
+    marker = mcg.contiguous()
+    return cls(
+        config=None,
+        in_features=int(suh.numel()),
+        out_features=int(svh.numel()),
+        trellis=trellis.contiguous(),
+        suh=suh.contiguous(),
+        svh=svh.contiguous(),
+        mcg=None if mul1 else marker,
+        mul1=marker if mul1 else None,
+        out_dtype=out_dtype,
+        transformers_fix=True,
+    )''')
+rep('''        if self.codebook != "mcg":
+            raise ValueError(
+                f"this overlay only implements codebook=mcg; got {self.codebook!r}"
+            )''','''        if self.codebook not in ("mcg", "mul1", "mixed", "auto"):
+            raise ValueError(
+                f"this overlay implements codebook mcg/mul1 (per layer); got {self.codebook!r}"
+            )''')
+rep('''        w13_mcg = Parameter(
+            torch.empty(num_experts, 2, 1, dtype=torch.int32),
+            requires_grad=False,
+        )''','''        w13_mcg = Parameter(
+            torch.zeros(num_experts, 2, 1, dtype=torch.int32),
+            requires_grad=False,
+        )
+        w13_mul1 = Parameter(
+            torch.zeros(num_experts, 2, 1, dtype=torch.int32),
+            requires_grad=False,
+        )''')
+rep('''        w2_mcg = Parameter(
+            torch.empty(num_experts, 1, dtype=torch.int32),
+            requires_grad=False,
+        )''','''        w2_mcg = Parameter(
+            torch.zeros(num_experts, 1, dtype=torch.int32),
+            requires_grad=False,
+        )
+        w2_mul1 = Parameter(
+            torch.zeros(num_experts, 1, dtype=torch.int32),
+            requires_grad=False,
+        )''')
+rep('''            "w13_mcg": w13_mcg,
+            "w2_trellis": w2_trellis,''','''            "w13_mcg": w13_mcg,
+            "w13_mul1": w13_mul1,
+            "w2_trellis": w2_trellis,''')
+rep('''            "w2_mcg": w2_mcg,
+        }''','''            "w2_mcg": w2_mcg,
+            "w2_mul1": w2_mul1,
+        }''')
+rep('''            "w13_mcg",
+            "w2_trellis",
+            "w2_suh",
+            "w2_svh",
+            "w2_mcg",
+        ):
+            getattr(layer, name)._exl3_owner = layer
+
+        mcg13 = layer.w13_mcg.reshape(-1)
+        mcg2 = layer.w2_mcg.reshape(-1)
+        if not torch.all(mcg13 == MCG_MARKER_SIGNED_INT32) or not torch.all(
+            mcg2 == MCG_MARKER_SIGNED_INT32
+        ):
+            raise RuntimeError(
+                "EXL3 mcg marker is not the MCG int32 0xCBAC1FED / "
+                f"{MCG_MARKER_SIGNED_INT32}; packed ABI mismatch"
+            )
+''','''            "w13_mcg",
+            "w13_mul1",
+            "w2_trellis",
+            "w2_suh",
+            "w2_svh",
+            "w2_mcg",
+            "w2_mul1",
+        ):
+            getattr(layer, name)._exl3_owner = layer
+
+        # KEYS PATCH: per-layer codebook detection from the loaded marker tensors.
+        mcg13 = layer.w13_mcg.reshape(-1)
+        mcg2 = layer.w2_mcg.reshape(-1)
+        mul13 = layer.w13_mul1.reshape(-1)
+        mul2 = layer.w2_mul1.reshape(-1)
+        is_mcg = bool(torch.all(mcg13 == MCG_MARKER_SIGNED_INT32)) and bool(
+            torch.all(mcg2 == MCG_MARKER_SIGNED_INT32)
+        )
+        is_mul1 = bool(torch.all(mul13 == MUL1_MARKER_SIGNED_INT32)) and bool(
+            torch.all(mul2 == MUL1_MARKER_SIGNED_INT32)
+        )
+        if is_mcg == is_mul1:
+            raise RuntimeError(
+                "EXL3 codebook markers inconsistent for this layer: "
+                f"mcg={is_mcg} mul1={is_mul1} (expect exactly one; "
+                f"MCG {MCG_MARKER_SIGNED_INT32} / MUL1 {MUL1_MARKER_SIGNED_INT32})"
+            )
+        use_mul1 = is_mul1
+        layer._exl3_mul1 = use_mul1
+        m13 = layer.w13_mul1 if use_mul1 else layer.w13_mcg
+        m2 = layer.w2_mul1 if use_mul1 else layer.w2_mcg
+        if not getattr(Exl3MoEMethod, "_logged_codebooks", None):
+            Exl3MoEMethod._logged_codebooks = set()
+        if use_mul1 not in Exl3MoEMethod._logged_codebooks:
+            Exl3MoEMethod._logged_codebooks.add(use_mul1)
+            logger.info("EXL3 experts codebook detected: %s", "mul1" if use_mul1 else "mcg")
+''')
+rep('''            gate = make_linear_exl3(
+                layer.w13_trellis[e, 0],
+                layer.w13_suh[e, 0],
+                layer.w13_svh[e, 0],
+                layer.w13_mcg[e, 0],
+            )
+            up = make_linear_exl3(
+                layer.w13_trellis[e, 1],
+                layer.w13_suh[e, 1],
+                layer.w13_svh[e, 1],
+                layer.w13_mcg[e, 1],
+            )
+            down = make_linear_exl3(
+                layer.w2_trellis[e],
+                layer.w2_suh[e],
+                layer.w2_svh[e],
+                layer.w2_mcg[e],
+            )''','''            gate = make_linear_exl3(
+                layer.w13_trellis[e, 0],
+                layer.w13_suh[e, 0],
+                layer.w13_svh[e, 0],
+                m13[e, 0],
+                mul1=use_mul1,
+            )
+            up = make_linear_exl3(
+                layer.w13_trellis[e, 1],
+                layer.w13_suh[e, 1],
+                layer.w13_svh[e, 1],
+                m13[e, 1],
+                mul1=use_mul1,
+            )
+            down = make_linear_exl3(
+                layer.w2_trellis[e],
+                layer.w2_suh[e],
+                layer.w2_svh[e],
+                m2[e],
+                mul1=use_mul1,
+            )''')
+rep('''        ptrs["down_svh"],
+        True,
+        False,
+        True,
+        False,
+        True,
+        False,
+        float(limit),''','''        ptrs["down_svh"],
+        # KEYS PATCH: (mcg, mul1) flag pairs for gate / up / down, per-layer codebook.
+        not _m1,
+        _m1,
+        not _m1,
+        _m1,
+        not _m1,
+        _m1,
+        float(limit),''')
+rep('''    k = int(getattr(layer, "_exl3_k", 4))
+    args = (''','''    k = int(getattr(layer, "_exl3_k", 4))
+    _m1 = bool(getattr(layer, "_exl3_mul1", False))
+    args = (''')
+open(dst,"w").write(s); print("overlay patched")

serving/patch_tp4_launcher.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+"""Add LOCAL_MODEL_DIR / LOCAL_DFLASH_DIR support to Mia's start-tp4.sh (serve a local checkpoint dir,
+e.g. an NFS path visible on all four nodes, instead of the HF-cache layout). Idempotent.
+Usage: patch_tp4_launcher.py ~/glm53-mia/start-tp4.sh   (then set LOCAL_MODEL_DIR=... in .env.tp4 and run with
+SKIP_DOWNLOAD=1 SKIP_SYNC=1 SKIP_PULL=1)"""
+import sys
+p = sys.argv[1]
+s = open(p).read()
+if "LOCAL_MODEL_DIR" in s:
+    print("already patched"); sys.exit(0)
+
+# workers: extend the per-rank preload string right before each rank's docker run
+old_w = '''        worker_ssh_n "$r" "docker run -d --name '$cname' \\'''
+new_w = '''        [ -n "${LOCAL_MODEL_DIR:-}" ] && worker_preload="$worker_preload -v '$LOCAL_MODEL_DIR:/model:ro'"
+        [ -n "${LOCAL_DFLASH_DIR:-}" ] && worker_preload="$worker_preload -v '$LOCAL_DFLASH_DIR:/draft:ro'"
+        worker_ssh_n "$r" "docker run -d --name '$cname' \\'''
+assert s.count(old_w) == 1, s.count(old_w)
+s = s.replace(old_w, new_w, 1)
+
+# head
+old_h = '''    docker run -d --name "$CONTAINER_HEAD" \\'''
+new_h = '''    [ -n "${LOCAL_MODEL_DIR:-}" ] && head_preload+=(-v "$LOCAL_MODEL_DIR:/model:ro")
+    [ -n "${LOCAL_DFLASH_DIR:-}" ] && head_preload+=(-v "$LOCAL_DFLASH_DIR:/draft:ro")
+    docker run -d --name "$CONTAINER_HEAD" \\'''
+assert s.count(old_h) == 1, s.count(old_h)
+s = s.replace(old_h, new_h, 1)
+
+# in-container model paths
+old_m = '''    MODEL_DIR="$(resolve_model_dir)"
+    DFLASH_MODEL_DIR=""
+    if [ "$SPEC_METHOD" = "dflash" ]; then
+        DFLASH_MODEL_DIR="$(resolve_dflash_dir)"'''
+new_m = '''    if [ -n "${LOCAL_MODEL_DIR:-}" ]; then
+        MODEL_DIR="/model"
+        log "LOCAL_MODEL_DIR mode: $LOCAL_MODEL_DIR -> /model on all ranks"
+    else
+        MODEL_DIR="$(resolve_model_dir)"
+    fi
+    DFLASH_MODEL_DIR=""
+    if [ "$SPEC_METHOD" = "dflash" ]; then
+        if [ -n "${LOCAL_DFLASH_DIR:-}" ]; then
+            DFLASH_MODEL_DIR="/draft"
+        else
+            DFLASH_MODEL_DIR="$(resolve_dflash_dir)"
+        fi'''
+assert s.count(old_m) == 1, s.count(old_m)
+s = s.replace(old_m, new_m, 1)
+open(p, "w").write(s)
+print("patched", p)

serving/sm120_dcp.py

@@ -0,0 +1,284 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""SM120 implementation variant for ``FLASHINFER_MLA_SPARSE_SM120``.
+
+keys-DCP port (2026-09-01): adds Decode Context Parallel (KV sharded across the TP ranks)
+support, mirroring the SM100 ``FlashInferMLASparseImpl`` branch:
+  * ``can_return_lse_for_decode = True`` so the CP compatibility check passes and the layer
+    combines per-rank partial attention with ``MLADCPManager.combine``;
+  * under DCP the global top-k token indices are filtered to this rank's local KV slots with
+    ``triton_filter_and_convert_dcp_index`` (interleave-aware);
+  * the flashinfer SM120 sparse kernel is asked for the softmax LSE (``return_lse``), which is
+    normalized to ``(num_tokens, num_heads)``; rows with no local top-k tokens get output 0 and
+    LSE ``-inf`` so they drop out of the cross-rank softmax merge;
+  * the output is allocated from ``q.shape[1]`` (all heads after the DCP query all-gather),
+    not the layer's TP-local head count.
+``EXL3_SM120_LSE_BASE_E`` (default 1) selects the LSE base reported to the combine kernel; the
+SM120 sparse kernel follows the FlashMLA (natural-log) convention. Set 0 for base-2.
+"""
+
+import os
+from typing import TYPE_CHECKING, cast
+
+import torch
+
+from vllm.v1.attention.backend import (
+    AttentionLayer,
+    AttentionType,
+    MLAAttentionImpl,
+)
+from vllm.v1.attention.backends.mla.flashinfer_mla_sparse import (
+    FlashInferMLASparseMetadata,
+    _get_workspace_buffer,
+)
+from vllm.v1.attention.backends.mla.sparse_utils import (
+    triton_convert_req_index_to_global_index,
+    triton_filter_and_convert_dcp_index,
+)
+
+if TYPE_CHECKING:
+    from vllm.model_executor.models.deepseek_v2 import Indexer
+
+
+# The layer's DCP combine reads ``attn_metadata.decode``; the sparse metadata has no such field.
+if not hasattr(FlashInferMLASparseMetadata, "decode"):
+    FlashInferMLASparseMetadata.decode = None  # type: ignore[attr-defined]
+
+
+def _kv_scale_format_for_model(model_type: str | None) -> str:
+    if model_type is not None and model_type.startswith("glm"):
+        return "arbitrary_fp32"
+    return "pow2_fp32"
+
+
+class FlashInferMLASparseSM120Impl(MLAAttentionImpl[FlashInferMLASparseMetadata]):
+    """SM120 FlashInfer sparse-MLA implementation (+ DCP)."""
+
+    is_sparse = True
+    supports_dense_mha_prefill = False
+    # DCP: we can hand the softmax LSE to the cross-rank combine.
+    can_return_lse_for_decode = True
+    lse_base_on_e = os.environ.get("EXL3_SM120_LSE_BASE_E", "1") == "1"
+
+    def __init__(
+        self,
+        num_heads: int,
+        head_size: int,
+        scale: float,
+        num_kv_heads: int,
+        alibi_slopes: list[float] | None,
+        sliding_window: int | None,
+        kv_cache_dtype: str,
+        logits_soft_cap: float | None,
+        attn_type: str,
+        kv_sharing_target_layer_name: str | None,
+        indexer: "Indexer | None" = None,
+        **mla_args,
+    ) -> None:
+        if any([alibi_slopes, sliding_window, logits_soft_cap]):
+            raise NotImplementedError(
+                "FLASHINFER_MLA_SPARSE_SM120 does not support alibi_slopes / "
+                "sliding_window / logits_soft_cap"
+            )
+        if attn_type != AttentionType.DECODER:
+            raise NotImplementedError(
+                "FLASHINFER_MLA_SPARSE_SM120 only supports decoder self-attention"
+            )
+
+        self.num_heads = num_heads
+        self.head_size = head_size
+        self.scale = float(scale)
+        self.num_kv_heads = num_kv_heads
+        self.kv_cache_dtype = kv_cache_dtype
+        if self.kv_cache_dtype != "fp8_ds_mla":
+            raise NotImplementedError(
+                "FLASHINFER_MLA_SPARSE_SM120 requires the packed fp8_ds_mla "
+                f"KV cache layout; got kv_cache_dtype={kv_cache_dtype!r}."
+            )
+
+        self.kv_lora_rank: int = mla_args["kv_lora_rank"]
+        self.qk_nope_head_dim: int = mla_args["qk_nope_head_dim"]
+        self.qk_rope_head_dim: int = mla_args["qk_rope_head_dim"]
+        self.rope_pad = 0
+        if self.qk_rope_head_dim == 0:
+            if self.kv_lora_rank != 512:
+                raise NotImplementedError(
+                    "FLASHINFER_MLA_SPARSE_SM120 pads NoPE MLA into the "
+                    "576-wide GLM_NSA geometry, which requires "
+                    f"kv_lora_rank=512; got {self.kv_lora_rank}."
+                )
+            self.rope_pad = 64
+        self.kernel_qk_rope_head_dim = self.qk_rope_head_dim + self.rope_pad
+        from vllm.config import get_current_vllm_config
+
+        vllm_config = get_current_vllm_config()
+        model_type = None
+        if vllm_config.model_config is not None:
+            model_type = getattr(
+                vllm_config.model_config.hf_text_config, "model_type", None
+            )
+        self.kv_scale_format = _kv_scale_format_for_model(model_type)
+
+        # Skip-topk layers are built with indexer=None and get the shared
+        # buffer via mla_args instead (cf. FLASHMLA_SPARSE).
+        self.topk_indices_buffer: torch.Tensor | None = (
+            indexer.topk_indices_buffer
+            if indexer is not None
+            else mla_args.get("topk_indices_buffer")
+        )
+        from vllm.utils.flashinfer import has_flashinfer_sparse_mla_sm120
+
+        if not has_flashinfer_sparse_mla_sm120():
+            raise RuntimeError(
+                "FLASHINFER_MLA_SPARSE_SM120 requires FlashInfer's "
+                "sparse MLA decode API."
+            )
+        assert self.topk_indices_buffer is not None
+
+        self.supports_quant_query_input = False
+        self._workspace_buffer: torch.Tensor | None = None
+        self._dcp_logged = False
+
+    @staticmethod
+    def _normalize_lse(lse: torch.Tensor, num_tokens: int, num_heads: int) -> torch.Tensor:
+        # (num_tokens, num_heads) is what the shared DCP reducer expects; the kernel may
+        # hand back (T, H, 1) / (T, 1, H) / (B, q_len, H).
+        if lse.dim() == 3:
+            if lse.shape[-1] == 1:
+                lse = lse.squeeze(-1)
+            elif lse.shape[1] == 1:
+                lse = lse.squeeze(1)
+            elif lse.shape[0] * lse.shape[1] == num_tokens:
+                lse = lse.reshape(num_tokens, lse.shape[-1])
+        if lse.shape != (num_tokens, num_heads):
+            raise RuntimeError(
+                "Unexpected SM120 sparse MLA LSE shape: "
+                f"{tuple(lse.shape)}, expected ({num_tokens}, {num_heads})."
+            )
+        return lse
+
+    def forward_mqa(
+        self,
+        q: torch.Tensor | tuple[torch.Tensor, torch.Tensor],
+        kv_c_and_k_pe_cache: torch.Tensor,
+        attn_metadata: FlashInferMLASparseMetadata,
+        layer: AttentionLayer,
+    ) -> tuple[torch.Tensor, torch.Tensor | None]:
+        if isinstance(q, tuple):
+            q = torch.cat(q, dim=-1)
+        if self.rope_pad:
+            q = torch.nn.functional.pad(q, (0, self.rope_pad))
+
+        num_actual_toks = q.shape[0]
+        # Under DCP the layer all-gathers queries across the CP ranks first, so q carries
+        # every head (not just this rank's TP slice).
+        num_heads = q.shape[1]
+
+        assert self.topk_indices_buffer is not None
+        topk_indices = self.topk_indices_buffer[:num_actual_toks]
+
+        use_dcp = self.dcp_world_size > 1
+        if use_dcp:
+            if not self._dcp_logged:
+                from vllm.logger import init_logger
+
+                init_logger(__name__).info(
+                    "[keys-DCP] SM120 sparse MLA: dcp_world=%d rank=%d interleave=%d "
+                    "heads=%d lse_base_e=%s",
+                    self.dcp_world_size,
+                    self.dcp_rank,
+                    attn_metadata.cp_kv_cache_interleave_size,
+                    num_heads,
+                    self.lse_base_on_e,
+                )
+                self._dcp_logged = True
+            topk_indices_physical, topk_lengths = cast(
+                tuple[torch.Tensor, torch.Tensor],
+                triton_filter_and_convert_dcp_index(
+                    attn_metadata.req_id_per_token[:num_actual_toks],
+                    attn_metadata.block_table,
+                    topk_indices,
+                    dcp_size=self.dcp_world_size,
+                    dcp_rank=self.dcp_rank,
+                    cp_kv_cache_interleave_size=attn_metadata.cp_kv_cache_interleave_size,
+                    BLOCK_SIZE=attn_metadata.block_size,
+                    NUM_TOPK_TOKENS=topk_indices.shape[1],
+                    return_valid_counts=True,
+                ),
+            )
+        else:
+            topk_indices_physical, topk_lengths = cast(
+                tuple[torch.Tensor, torch.Tensor],
+                triton_convert_req_index_to_global_index(
+                    attn_metadata.req_id_per_token[:num_actual_toks],
+                    attn_metadata.block_table,
+                    topk_indices,
+                    BLOCK_SIZE=attn_metadata.block_size,
+                    NUM_TOPK_TOKENS=topk_indices.shape[1],
+                    return_valid_counts=True,
+                ),
+            )
+        sparse_topk_capacity = topk_indices_physical.shape[1]
+        empty_rows = topk_lengths == 0
+        topk_indices_physical[:, 0] = topk_indices_physical[:, 0].masked_fill(
+            empty_rows, 0
+        )
+        topk_lengths = topk_lengths.clamp(min=1)
+
+        output = q.new_empty(
+            (num_actual_toks, num_heads, self.kv_lora_rank),
+            dtype=q.dtype,
+        )
+
+        if self._workspace_buffer is None:
+            self._workspace_buffer = _get_workspace_buffer(q.device)
+
+        from vllm.utils.flashinfer import (
+            flashinfer_trtllm_batch_decode_with_kv_cache_mla,
+        )
+
+        want_lse = bool(self.need_to_return_lse_for_decode)
+        kernel_out = flashinfer_trtllm_batch_decode_with_kv_cache_mla(
+            query=q.unsqueeze(1),
+            kv_cache=kv_c_and_k_pe_cache.view(torch.uint8).unsqueeze(1),
+            workspace_buffer=self._workspace_buffer,
+            qk_nope_head_dim=self.qk_nope_head_dim,
+            kv_lora_rank=self.kv_lora_rank,
+            qk_rope_head_dim=self.kernel_qk_rope_head_dim,
+            block_tables=topk_indices_physical.unsqueeze(1),
+            seq_lens=topk_lengths,
+            max_seq_len=sparse_topk_capacity,
+            out=output.unsqueeze(1),
+            bmm1_scale=self.scale,
+            bmm2_scale=1.0,
+            sparse_mla_top_k=sparse_topk_capacity,
+            kv_scale_format=self.kv_scale_format,
+            return_lse=want_lse,
+        )
+        lse: torch.Tensor | None = None
+        if want_lse:
+            assert isinstance(kernel_out, tuple), "SM120 sparse MLA did not return LSE"
+            out, lse = kernel_out
+        else:
+            out = kernel_out
+        out = out.view(-1, out.shape[-2], out.shape[-1])
+        out.masked_fill_(empty_rows.view(-1, 1, 1), 0.0)
+        if lse is not None:
+            lse = self._normalize_lse(lse, out.shape[0], out.shape[1])
+            lse = lse.masked_fill(empty_rows.view(-1, 1), float("-inf"))
+        return out, lse
+
+    def do_kv_cache_update(
+        self,
+        kv_c_normed: torch.Tensor,
+        k_pe: torch.Tensor,
+        kv_cache: torch.Tensor,
+        slot_mapping: torch.Tensor,
+        kv_cache_dtype: str,
+        k_scale: torch.Tensor,
+    ) -> None:
+        if self.rope_pad:
+            k_pe = k_pe.new_zeros((k_pe.shape[0], 1, self.rope_pad))
+        super().do_kv_cache_update(
+            kv_c_normed, k_pe, kv_cache, slot_mapping, kv_cache_dtype, k_scale
+        )

serving/start-tp4.sh

@@ -0,0 +1,1776 @@
+#!/usr/bin/env bash
+# ============================================================================
+# start-tp4.sh — EXPERIMENTAL 4× DGX Spark TP=4 launcher for GLM-5.3-Flash EXL3
+# ============================================================================
+#
+# Optional sibling of start.sh. Does not change the supported 2× TP=2 path.
+# Untested here (no 4-Spark kit). A user with four GB10s can try:
+#
+#   ./start-tp4.sh
+#
+# Layout (mp executor, not Ray):
+#   rank 0  HEAD_IP     (default 10.0.0.1)  — vLLM API on :8888
+#   rank 1  WORKER_IP   (default 10.0.0.2)  — --headless
+#   rank 2  WORKER2_IP  (default 10.0.0.3)  — --headless
+#   rank 3  WORKER3_IP  (default 10.0.0.4)  — --headless
+#   --tensor-parallel-size 4  --nnodes 4
+#
+# Fabric: four Sparks need a RoCE path among all ranks (QSFP ring using both
+# CX7 ports, or a switch). Defaults set NCCL_CROSS_NIC=1. Pin per-rank
+# WORKER2_CX7_IF / WORKER2_CX7_IB / WORKER2_GID (same for 3) in .env.
+# Dual-port example: RANK NCCL_IB_HCA="rocep1s0f0,rocep1s0f1".
+#
+# Same image/weights as start.sh. Container names are glm53-exl3-tp4-* so a
+# TP=2 serve is not accidentally reused. Stop with ./start-tp4.sh stop
+# (./start.sh stop does not know about ranks 2/3).
+#
+# Usage:
+#   ./start-tp4.sh                 start TP=4
+#   ./start-tp4.sh download        head HF cache only
+#   ./start-tp4.sh stop|restart|status
+#   ./start-tp4.sh logs            follow head
+#   ./start-tp4.sh logs 1|2|3      follow that worker rank
+#
+# Extra knobs live in .env.tp4 (copied from .env.tp4.example). start.sh
+# never reads that file. Shared tokens/IPs can stay in .env.
+# ============================================================================
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
+
+if [ ! -f "$SCRIPT_DIR/.env" ]; then
+    [ -f "$SCRIPT_DIR/.env.example" ] || {
+        echo "ERROR: missing .env.example" >&2
+        exit 1
+    }
+    cp "$SCRIPT_DIR/.env.example" "$SCRIPT_DIR/.env"
+    printf '\033[1;36m[glm53-exl3-tp4]\033[0m wrote .env from .env.example\n'
+fi
+if [ ! -f "$SCRIPT_DIR/.env.tp4" ]; then
+    [ -f "$SCRIPT_DIR/.env.tp4.example" ] || {
+        echo "ERROR: missing .env.tp4.example" >&2
+        exit 1
+    }
+    cp "$SCRIPT_DIR/.env.tp4.example" "$SCRIPT_DIR/.env.tp4"
+    printf '\033[1;36m[glm53-exl3-tp4]\033[0m wrote .env.tp4 from .env.tp4.example — edit WORKER2_IP / WORKER3_IP / CX7 pins\n'
+fi
+# Caller exports (MTP_TOKENS=2 ./start.sh restart) must win over .env.
+_cli_mtp="${MTP_TOKENS-}"
+_cli_spec="${SPEC_METHOD-}"
+_cli_eager="${ENFORCE_EAGER-}"
+_cli_fused="${EXL3_FUSED_MOE-}"
+_cli_row_tile="${EXL3_MOE_ROW_TILE-}"
+_cli_temp_rows="${EXL3_TEMP_ROWS_FUSED-}"
+_cli_fat_sorted="${EXL3_FAT_SORTED-}"
+_cli_fat_batched="${EXL3_FAT_BATCHED-}"
+_cli_fat_kernel="${EXL3_FAT_KERNEL-}"
+_cli_mnbt="${MAX_NUM_BATCHED_TOKENS-}"
+_cli_image="${IMAGE-}"
+_cli_util="${GPU_MEM_UTIL-}"
+_cli_lm="${LANGUAGE_MODEL_ONLY-}"
+_cli_max_num_seqs="${MAX_NUM_SEQS-}"
+_cli_ablit="${ABLIT-}"
+_cli_ablit_method="${ABLIT_METHOD-}"
+_cli_ablit_direction="${ABLIT_DIRECTION-}"
+_cli_ablit_layers="${ABLIT_LAYERS-}"
+_cli_ablit_alpha="${ABLIT_ALPHA-}"
+_cli_ablit_mtp="${ABLIT_INCLUDE_MTP-}"
+# Setness-aware: an explicitly empty caller value is an operator error and
+# must reach validate_numeric_config, not be swallowed by a .env value.
+_cli_indexer_workspace_set="${GLM53_INDEXER_WORKSPACE+1}"
+_cli_indexer_workspace="${GLM53_INDEXER_WORKSPACE-}"
+_cli_spinwait_ms_set="${GLM53_SPINWAIT_MS+1}"
+_cli_spinwait_ms="${GLM53_SPINWAIT_MS-}"
+set -a
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/.env"
+# TP=4 overlay wins over the 2× knobs in .env.
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/.env.tp4"
+set +a
+[ -n "${_cli_mtp}" ] && MTP_TOKENS="$_cli_mtp"
+[ -n "${_cli_spec}" ] && SPEC_METHOD="$_cli_spec"
+[ -n "${_cli_eager}" ] && ENFORCE_EAGER="$_cli_eager"
+[ -n "${_cli_fused}" ] && EXL3_FUSED_MOE="$_cli_fused"
+[ -n "${_cli_row_tile}" ] && EXL3_MOE_ROW_TILE="$_cli_row_tile"
+[ -n "${_cli_temp_rows}" ] && EXL3_TEMP_ROWS_FUSED="$_cli_temp_rows"
+[ -n "${_cli_fat_sorted}" ] && EXL3_FAT_SORTED="$_cli_fat_sorted"
+[ -n "${_cli_fat_batched}" ] && EXL3_FAT_BATCHED="$_cli_fat_batched"
+[ -n "${_cli_fat_kernel}" ] && EXL3_FAT_KERNEL="$_cli_fat_kernel"
+[ -n "${_cli_mnbt}" ] && MAX_NUM_BATCHED_TOKENS="$_cli_mnbt"
+[ -n "${_cli_image}" ] && IMAGE="$_cli_image"
+[ -n "${_cli_util}" ] && GPU_MEM_UTIL="$_cli_util"
+[ -n "${_cli_lm}" ] && LANGUAGE_MODEL_ONLY="$_cli_lm"
+[ -n "${_cli_max_num_seqs}" ] && MAX_NUM_SEQS="$_cli_max_num_seqs"
+[ -n "${_cli_ablit}" ] && ABLIT="$_cli_ablit"
+[ -n "${_cli_ablit_method}" ] && ABLIT_METHOD="$_cli_ablit_method"
+[ -n "${_cli_ablit_direction}" ] && ABLIT_DIRECTION="$_cli_ablit_direction"
+[ -n "${_cli_ablit_layers}" ] && ABLIT_LAYERS="$_cli_ablit_layers"
+[ -n "${_cli_ablit_alpha}" ] && ABLIT_ALPHA="$_cli_ablit_alpha"
+[ -n "${_cli_ablit_mtp}" ] && ABLIT_INCLUDE_MTP="$_cli_ablit_mtp"
+[ -n "${_cli_indexer_workspace_set}" ] && GLM53_INDEXER_WORKSPACE="$_cli_indexer_workspace"
+[ -n "${_cli_spinwait_ms_set}" ] && GLM53_SPINWAIT_MS="$_cli_spinwait_ms"
+
+# ----------------------------- configuration -------------------------------
+MODEL="${MODEL:-Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw}"
+# If the durable mirror is empty/moved, download.sh falls back to this id.
+MODEL_FALLBACK="${MODEL_FALLBACK:-brandonmusic/GLM-5.3-Flash-tr3-4bpw}"
+MODEL_CACHE_NAME="${MODEL_CACHE_NAME:-models--${MODEL//\//--}}"
+MODEL_FALLBACK_CACHE_NAME="${MODEL_FALLBACK_CACHE_NAME:-models--${MODEL_FALLBACK//\//--}}"
+# Hub commit on the Mia-AiLab mirror (the 5ab363a8-byte-identical upload).
+MODEL_REVISION="${MODEL_REVISION:-25a44fdbf16862a46b7cc9921142c6c81350af2f}"
+IMAGE="${IMAGE:-ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3}"
+SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-GLM-5.3-Flash-EXL3}"
+GHCR_USER="${GHCR_USER:-MiaAI-Lab}"
+
+HEAD_IP="${HEAD_IP:-10.0.0.1}"
+WORKER_IP="${WORKER_IP:-10.0.0.2}"
+# Same OS user on both Sparks unless .env sets WORKER_USER (mixed-account kits).
+WORKER_USER="${WORKER_USER:-$USER}"
+if [ "$WORKER_USER" = "$USER" ]; then
+    WORKER_HOME="${WORKER_HOME:-$HOME}"
+else
+    WORKER_HOME="${WORKER_HOME:-/home/${WORKER_USER}}"
+fi
+WORKER_SSH="${WORKER_SSH:-${WORKER_USER}@${WORKER_IP}}"
+
+# Ranks 2 and 3 (experimental TP=4). Same OS user as WORKER_USER unless set.
+WORKER2_IP="${WORKER2_IP:-10.0.0.3}"
+WORKER3_IP="${WORKER3_IP:-10.0.0.4}"
+WORKER2_USER="${WORKER2_USER:-$WORKER_USER}"
+WORKER3_USER="${WORKER3_USER:-$WORKER_USER}"
+if [ "$WORKER2_USER" = "$USER" ]; then
+    WORKER2_HOME="${WORKER2_HOME:-$HOME}"
+else
+    WORKER2_HOME="${WORKER2_HOME:-/home/${WORKER2_USER}}"
+fi
+if [ "$WORKER3_USER" = "$USER" ]; then
+    WORKER3_HOME="${WORKER3_HOME:-$HOME}"
+else
+    WORKER3_HOME="${WORKER3_HOME:-/home/${WORKER3_USER}}"
+fi
+WORKER2_SSH="${WORKER2_SSH:-${WORKER2_USER}@${WORKER2_IP}}"
+WORKER3_SSH="${WORKER3_SSH:-${WORKER3_USER}@${WORKER3_IP}}"
+
+HEAD_CX7_IF="${HEAD_CX7_IF:-enp1s0f1np1}"
+WORKER_CX7_IF="${WORKER_CX7_IF:-enp1s0f0np0}"
+HEAD_CX7_IB="${HEAD_CX7_IB:-rocep1s0f1}"
+WORKER_CX7_IB="${WORKER_CX7_IB:-rocep1s0f0}"
+WORKER2_CX7_IF="${WORKER2_CX7_IF:-$WORKER_CX7_IF}"
+WORKER2_CX7_IB="${WORKER2_CX7_IB:-$WORKER_CX7_IB}"
+WORKER3_CX7_IF="${WORKER3_CX7_IF:-$WORKER_CX7_IF}"
+WORKER3_CX7_IB="${WORKER3_CX7_IB:-$WORKER_CX7_IB}"
+NCCL_DEBUG="${NCCL_DEBUG:-WARN}"
+NCCL_IB_GID_INDEX="${NCCL_IB_GID_INDEX:-3}"
+# The RoCEv2 GID index is per-NIC: the usable entry is the one whose GID matches
+# that node's own fabric IP. Most pairs share a good index; some do not (this kit
+# needs head=4, worker=3). Unset, both inherit NCCL_IB_GID_INDEX -> unchanged.
+HEAD_GID="${HEAD_GID:-$NCCL_IB_GID_INDEX}"
+WORKER_GID="${WORKER_GID:-$NCCL_IB_GID_INDEX}"
+WORKER2_GID="${WORKER2_GID:-$NCCL_IB_GID_INDEX}"
+WORKER3_GID="${WORKER3_GID:-$NCCL_IB_GID_INDEX}"
+# vLLM subtracts a CUDA-graph memory ESTIMATE from the KV pool. On this kit the
+# estimate is 2.43 GiB while the captured graphs actually consume -0.19 GiB, so
+# ~2.6 GiB of KV is reserved and never used. 0 keeps CUDA graphs ON and drops only
+# the deduction. 1 = upstream default.
+CG_ESTIMATE="${CG_ESTIMATE:-1}"
+NCCL_CROSS_NIC="${NCCL_CROSS_NIC:-1}"
+NCCL_HOST_DIR="${NCCL_HOST_DIR:-$HOME/nccl-2.30.7}"
+WORKER_NCCL_HOST_DIR="${WORKER_NCCL_HOST_DIR:-$WORKER_HOME/nccl-2.30.7}"
+WORKER2_NCCL_HOST_DIR="${WORKER2_NCCL_HOST_DIR:-$WORKER2_HOME/nccl-2.30.7}"
+WORKER3_NCCL_HOST_DIR="${WORKER3_NCCL_HOST_DIR:-$WORKER3_HOME/nccl-2.30.7}"
+NCCL_SO_NAME="${NCCL_SO_NAME:-libnccl.so.2.30.7}"
+# glm53-flash already ships nvidia-nccl. LD_PRELOAD of the host 2.30.7 SO
+# makes DeepEP assert duplicate NCCL (/nccl/... vs nvidia/nccl/lib/...).
+# Set USE_HOST_NCCL=1 only if image NCCL cannot talk CX7.
+USE_HOST_NCCL="${USE_HOST_NCCL:-0}"
+
+TP="${TP:-4}"
+NNODES="${NNODES:-4}"
+PORT="${PORT:-8888}"
+MASTER_PORT="${MASTER_PORT:-29521}"
+
+MTP_TOKENS="${MTP_TOKENS:-2}"
+# dflash (default, incoai/GLM-5.3-Flash-DFlash2, k=7) | mtp | none
+SPEC_METHOD="${SPEC_METHOD:-dflash}"
+DFLASH_MODEL="${DFLASH_MODEL:-incoai/GLM-5.3-Flash-DFlash2}"
+DFLASH_CACHE_NAME="${DFLASH_CACHE_NAME:-models--${DFLASH_MODEL//\//--}}"
+DFLASH_TOKENS="${DFLASH_TOKENS:-7}"
+# 2 = shard the ~2.3 GiB DFlash2 drafter across TP (C4 keep, 2026-08-30:
+# idle 8k 938 / 16k 972 / 100k 997; decode structured 65.1 / prose 27.1).
+# 1 = rank 0 only (no CX7 on every draft step). Empty = inherit target TP.
+# Do not pin attention_backend: SM121 already prefers FLASH_ATTN for
+# non-causal dense SWA. TRITON_ATTN was an SM120 mask-fix this image lacks.
+DFLASH_DRAFT_TP="${DFLASH_DRAFT_TP-4}"
+MAX_MODEL_LEN="${MAX_MODEL_LEN:-1000000}"
+GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.87}"
+MAX_NUM_SEQS="${MAX_NUM_SEQS:-4}"
+# 8192 chunk × long history oversubscribes GB10 persistent_topk smem (300k crash).
+# E2 one-shot 2026-09-01: 7168 keep (100k ~1148 / 300k ~1107); 2048/3548 similar or slower.
+MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-7168}"
+CHAT_TEMPLATE_HOST="${CHAT_TEMPLATE_HOST:-$SCRIPT_DIR/files/chat_template.jinja}"
+CHAT_TEMPLATE="${CHAT_TEMPLATE:-/opt/glm53/chat_template.jinja}"
+VIDEO_PATCH_HOST="${VIDEO_PATCH_HOST:-$SCRIPT_DIR/overlay/patch_glm_video_placeholders.py}"
+STOP_PATCH_HOST="${STOP_PATCH_HOST:-$SCRIPT_DIR/overlay/patch_suppress_stops_in_reasoning.py}"
+SCHED_PATCH_HOST="${SCHED_PATCH_HOST:-$SCRIPT_DIR/overlay/patch_scheduler_decode_floor.py}"
+DRAFTER_PATCH_HOST="${DRAFTER_PATCH_HOST:-$SCRIPT_DIR/overlay/patch_glm5_drafter_group.py}"
+APC_PATCH_HOST="${APC_PATCH_HOST:-$SCRIPT_DIR/overlay/patch_hybrid_prefix_hit.py}"
+XGRAMMAR_PATCH_HOST="${XGRAMMAR_PATCH_HOST:-$SCRIPT_DIR/overlay/patch_xgrammar_termination.py}"
+KPOOL_TAIL_PATCH_HOST="${KPOOL_TAIL_PATCH_HOST:-$SCRIPT_DIR/overlay/patch_kpool_tail_slotmap.py}"
+SPINWAIT_PATCH_HOST="${SPINWAIT_PATCH_HOST:-$SCRIPT_DIR/overlay/patch_spinwait.py}"
+KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}"
+QUANTIZATION="${QUANTIZATION:-exl3}"
+LANGUAGE_MODEL_ONLY="${LANGUAGE_MODEL_ONLY:-0}"
+SKIP_MM_PROFILING="${SKIP_MM_PROFILING:-1}"
+# JSON default cannot sit in ${LIMIT_MM:-{...}} — } ends the expansion.
+if [ -z "${LIMIT_MM:-}" ]; then
+    LIMIT_MM='{"image":4,"video":1}'
+fi
+TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST:-12.1a}"
+FLASHINFER_CUDA_ARCH_LIST="${FLASHINFER_CUDA_ARCH_LIST:-12.1a}"
+# Graph-safe fused apply (device-side expert grouping). MTP k=2 decode is
+# 1..4 seqs × 3 tokens (must include 3). DFlash2 k=7 is 1..4 seqs × 8 tokens
+# (must include 8, 16, 24, 32).
+ENFORCE_EAGER="${ENFORCE_EAGER:-0}"
+if [ "${ENFORCE_EAGER}" != "1" ]; then
+    case " ${EXTRA_ARGS:-} " in
+        *" --cudagraph-capture-sizes "*|*" cudagraph-capture-sizes "*) ;;
+        *)
+            if [ "$SPEC_METHOD" = "dflash" ]; then
+                EXTRA_ARGS="${EXTRA_ARGS:+$EXTRA_ARGS }--cudagraph-capture-sizes 1 2 4 8 16 24 32"
+            else
+                EXTRA_ARGS="${EXTRA_ARGS:+$EXTRA_ARGS }--cudagraph-capture-sizes 1 2 3 4 6 8 12"
+            fi
+            ;;
+    esac
+fi
+# 1 = fused exl3_moe (decode). 0 restores the unique-expert LinearEXL3 loop.
+EXL3_FUSED_MOE="${EXL3_FUSED_MOE:-1}"
+# 1 = GPU row tiles for fat experts (prefill). 0 = LinearEXL3 fallback.
+# Tile (P2a) and TEMP_ROWS=1024 (P2b) both lost at MNBT=1024 — leave 128.
+EXL3_MOE_ROW_TILE="${EXL3_MOE_ROW_TILE:-0}"
+# Fused exl3_moe temp rows/expert. 1024 was slower than 128+fallback (P2b).
+EXL3_TEMP_ROWS_FUSED="${EXL3_TEMP_ROWS_FUSED:-128}"
+# Sorted routing tier; higher tiers imply it even when this is 0.
+EXL3_FAT_SORTED="${EXL3_FAT_SORTED:-0}"
+# E1 batched tier: persistent scratch + combined gate/up; implies SORTED=1.
+EXL3_FAT_BATCHED="${EXL3_FAT_BATCHED:-0}"
+# E2 direct trellis kernel (default on). Implies BATCHED=1 and SORTED=1.
+# Needs the patched extension — start.sh rebuilds when the recipe stamp drifts.
+# Set all three flags to 0 for the legacy fat-expert path.
+EXL3_FAT_KERNEL="${EXL3_FAT_KERNEL:-1}"
+
+# --- abliteration (ablit/) --------------------------------------------------
+# Load-time o_proj orthogonalization (overlay/ablit_runtime.py). Published
+# recipe: layers 15-45 edited with the dealign direction, 0-14 stay stock
+# safety anchors, MTP block included. 0 = stock weights. Applied identically
+# on both TP ranks; the DFlash2 drafter is never touched.
+ABLIT="${ABLIT:-0}"
+ABLIT_METHOD="${ABLIT_METHOD:-auto}"           # auto | transplant | proj
+ABLIT_DIRECTION="${ABLIT_DIRECTION:-dealign}"  # dealign | bf_oproj | /path/dir.pt
+ABLIT_LAYERS="${ABLIT_LAYERS:-15-45}"          # inclusive; 45 = checkpoint MTP block
+ABLIT_ALPHA="${ABLIT_ALPHA:-3.0}"              # 1.0 = plain projection, >1 over-projects
+ABLIT_INCLUDE_MTP="${ABLIT_INCLUDE_MTP:-1}"
+
+READY_TIMEOUT="${READY_TIMEOUT:-3600}"
+# 1 = suppress client stop strings until </think> (DSpark #42 class).
+GLM53_SUPPRESS_STOPS_IN_REASONING="${GLM53_SUPPRESS_STOPS_IN_REASONING:-1}"
+# Mixed-step prefill policy when a peer is already decoding (issue #6).
+# skip = do not mix; N>0 = cap tokens; 0 = off.
+GLM53_MIXED_PREFILL_CHUNK="${GLM53_MIXED_PREFILL_CHUNK:-skip}"
+# Sparse-indexer prefill gather workspace (overlay/patch_indexer_workspace.py).
+# stock = max_model_len * 40 entries (5036.40 MB locked at 1M, measured);
+# rightsize = the legal per-step maximum, ~+26% KV. Default applies only
+# when UNSET: an explicitly empty value is an operator error and
+# validate_numeric_config rejects it rather than guessing a serving mode.
+GLM53_INDEXER_WORKSPACE="${GLM53_INDEXER_WORKSPACE-stock}"
+# SpinCondition reader busy-loop window. "stock" preserves vLLM's 1 s default;
+# 1..1000 selects milliseconds. The frozen TP=2 sweep selected 16 ms.
+GLM53_SPINWAIT_MS="${GLM53_SPINWAIT_MS-stock}"
+# EngineCore stock timeout is 300s; mid-serve Triton/TileLang JIT on TP=2 can
+# exceed that without being a true hang. NCCL watchdog is still 600s.
+VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS="${VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS:-1800}"
+# 1 = after /health, burn DFlash2 BLOCK / sampler / kpool shapes. Nonfatal.
+GLM53_BOOT_SHAPE_WARMUP="${GLM53_BOOT_SHAPE_WARMUP:-1}"
+GLM53_WARMUP_REQ_TIMEOUT="${GLM53_WARMUP_REQ_TIMEOUT:-240}"
+
+# OpenAI-compatible API bearer token. Read the native VLLM_API_KEY env var
+# (vLLM falls back to it when --api-key is absent on the CLI), so the key
+# never lands in argv / `non-default args` startup log. Empty = no auth.
+# Same single-key semantics as the DeepSeek V4 Flash DSpark deployment.
+VLLM_API_KEY="${VLLM_API_KEY:-}"
+
+CONTAINER_HEAD="${CONTAINER_HEAD:-glm53-exl3-tp4-head}"
+CONTAINER_WORKER="${CONTAINER_WORKER:-glm53-exl3-tp4-w1}"
+CONTAINER_WORKER2="${CONTAINER_WORKER2:-glm53-exl3-tp4-w2}"
+CONTAINER_WORKER3="${CONTAINER_WORKER3:-glm53-exl3-tp4-w3}"
+
+HF_CACHE_DIR="${HF_HOME:-$HOME/.cache/huggingface}"
+MODEL_PATH="$HF_CACHE_DIR/hub/$MODEL_CACHE_NAME"
+FALLBACK_MODEL_PATH="$HF_CACHE_DIR/hub/$MODEL_FALLBACK_CACHE_NAME"
+DFLASH_PATH="$HF_CACHE_DIR/hub/$DFLASH_CACHE_NAME"
+WORKER_CACHE_DIR="$WORKER_HOME/.cache/huggingface"
+WORKER2_CACHE_DIR="${WORKER2_CACHE_DIR:-$WORKER2_HOME/.cache/huggingface}"
+WORKER3_CACHE_DIR="${WORKER3_CACHE_DIR:-$WORKER3_HOME/.cache/huggingface}"
+CACHE_ROOT="${CACHE_ROOT:-$HOME/.cache/vllm-glm53-flash}"
+WORKER_VLLM_CACHE="${WORKER_VLLM_CACHE:-$WORKER_HOME/.cache/vllm-glm53-flash}"
+WORKER2_VLLM_CACHE="${WORKER2_VLLM_CACHE:-$WORKER2_HOME/.cache/vllm-glm53-flash}"
+WORKER3_VLLM_CACHE="${WORKER3_VLLM_CACHE:-$WORKER3_HOME/.cache/vllm-glm53-flash}"
+# Overlay FS ~/.triton and ~/.tilelang die on container recreate (TP=2 JIT
+# stall → 600s NCCL watchdog). Persist next to the vLLM cache.
+TRITON_HOST_CACHE="${TRITON_HOST_CACHE:-$CACHE_ROOT/triton}"
+TILELANG_HOST_CACHE="${TILELANG_HOST_CACHE:-$CACHE_ROOT/tilelang}"
+WORKER_TRITON_CACHE="${WORKER_TRITON_CACHE:-$WORKER_VLLM_CACHE/triton}"
+WORKER_TILELANG_CACHE="${WORKER_TILELANG_CACHE:-$WORKER_VLLM_CACHE/tilelang}"
+WORKER2_TRITON_CACHE="${WORKER2_TRITON_CACHE:-$WORKER2_VLLM_CACHE/triton}"
+WORKER2_TILELANG_CACHE="${WORKER2_TILELANG_CACHE:-$WORKER2_VLLM_CACHE/tilelang}"
+WORKER3_TRITON_CACHE="${WORKER3_TRITON_CACHE:-$WORKER3_VLLM_CACHE/triton}"
+WORKER3_TILELANG_CACHE="${WORKER3_TILELANG_CACHE:-$WORKER3_VLLM_CACHE/tilelang}"
+TRITON_CACHE_DIR="${TRITON_CACHE_DIR:-/root/.triton/cache}"
+TILELANG_CACHE_DIR="${TILELANG_CACHE_DIR:-/root/.tilelang/cache}"
+
+LOGDIR="$SCRIPT_DIR/logs"
+HEAD_SCRIPT="$SCRIPT_DIR/.glm53-exl3-tp4-head.inner.sh"
+WORKER_SCRIPT="$SCRIPT_DIR/.glm53-exl3-tp4-worker.inner.sh"
+EXPECTED_SHARDS="${EXPECTED_SHARDS:-120}"
+
+# ------------------------------- helpers -----------------------------------
+log()  { printf '\033[1;36m[glm53-exl3-tp4]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[glm53-exl3-tp4]\033[0m %s\n' "$*" >&2; }
+die()  { printf '\033[1;31m[glm53-exl3-tp4]\033[0m ERROR: %s\n' "$*" >&2; exit 1; }
+
+# GLM53 numeric config guard (begin)
+_glm53_canonical_positive_int() {
+    local name="$1" value="$2" maximum="$3" canonical
+    if ! [[ "$value" =~ ^[0-9]+$ ]]; then
+        echo "$name must be a positive base-10 integer (got: $value)" >&2
+        return 2
+    fi
+    canonical="$value"
+    while [ "${canonical#0}" != "$canonical" ]; do canonical="${canonical#0}"; done
+    [ -n "$canonical" ] || canonical=0
+    if [ "$canonical" = 0 ] \
+       || [ "${#canonical}" -gt "${#maximum}" ] \
+       || [ "$canonical" -gt "$maximum" ]; then
+        echo "$name must be between 1 and $maximum (got: $value)" >&2
+        return 2
+    fi
+    printf -v "$name" '%s' "$canonical"
+    # $name is a validated integer configuration variable.
+    # shellcheck disable=SC2163
+    export "$name"
+}
+
+# Enum knobs are exactly one of a fixed set. Not "non-empty means on": a
+# typo'd knob must not silently pick a serving mode. GLM53_INDEXER_WORKSPACE
+# sizes the sparse-indexer prefill workspace, and the patched
+# get_max_prefill_buffer_size itself raises on anything but stock/rightsize
+# (overlay/patch_indexer_workspace.py, _glm53_workspace_mode), so catching it
+# here turns a container boot failure into a launcher error. The match is
+# literal on both sides -- the "-stock" default applies only to an UNSET var,
+# so "", " rightsize " and "RIGHTSIZE" all fail here and would fail there.
+_glm53_validate_enum() {
+    local name="$1" value="$2" allowed
+    shift 2
+    for allowed in "$@"; do
+        [ "$value" = "$allowed" ] && return 0
+    done
+    echo "$name must be one of: $* (got: $value)" >&2
+    return 2
+}
+
+_glm53_validate_spinwait_ms() {
+    if [ "$GLM53_SPINWAIT_MS" = "stock" ]; then
+        export GLM53_SPINWAIT_MS
+        return 0
+    fi
+    _glm53_canonical_positive_int \
+        GLM53_SPINWAIT_MS "$GLM53_SPINWAIT_MS" 1000
+}
+
+validate_numeric_config() {
+    if ! [[ "$GPU_MEM_UTIL" =~ ^(0([.][0-9]+)?|[.][0-9]+|1([.]0+)?)$ ]] \
+       || ! awk -v u="$GPU_MEM_UTIL" 'BEGIN { exit !(u > 0 && u <= 1) }'; then
+        echo "GPU_MEM_UTIL must be greater than 0 and at most 1 (got: $GPU_MEM_UTIL)" >&2
+        return 2
+    fi
+    _glm53_canonical_positive_int MAX_MODEL_LEN "$MAX_MODEL_LEN" 1000000 || return
+    _glm53_canonical_positive_int MAX_NUM_SEQS "$MAX_NUM_SEQS" 4096 || return
+    _glm53_canonical_positive_int MAX_NUM_BATCHED_TOKENS "$MAX_NUM_BATCHED_TOKENS" 8388608 || return
+    _glm53_validate_enum GLM53_INDEXER_WORKSPACE "${GLM53_INDEXER_WORKSPACE-stock}" \
+        stock rightsize || return
+    _glm53_validate_spinwait_ms || return
+}
+# GLM53 numeric config guard (end)
+
+banner() {
+    local label="${1:-start-tp4.sh}"
+    printf '\n'
+    printf '  \033[1;36m┌────────────────────────────────────────────┐\033[0m\n'
+    printf '  \033[1;36m│\033[0m  \033[1mGLM-5.3 Flash EXL3\033[0m  \033[2m·  %-11s\033[0m        \033[1;36m│\033[0m\n' "$label"
+    printf '  \033[1;36m└────────────────────────────────────────────┘\033[0m\n'
+    printf '\n'
+}
+
+# rank 1..3
+_tp4_ssh_target() {
+    case "$1" in
+        1) printf '%s' "$WORKER_SSH" ;;
+        2) printf '%s' "$WORKER2_SSH" ;;
+        3) printf '%s' "$WORKER3_SSH" ;;
+        *) die "internal: bad worker rank $1" ;;
+    esac
+}
+_tp4_rank_ip() {
+    case "$1" in
+        1) printf '%s' "$WORKER_IP" ;;
+        2) printf '%s' "$WORKER2_IP" ;;
+        3) printf '%s' "$WORKER3_IP" ;;
+    esac
+}
+_tp4_rank_home() {
+    case "$1" in
+        1) printf '%s' "$WORKER_HOME" ;;
+        2) printf '%s' "$WORKER2_HOME" ;;
+        3) printf '%s' "$WORKER3_HOME" ;;
+    esac
+}
+_tp4_rank_hf() {
+    case "$1" in
+        1) printf '%s' "$WORKER_CACHE_DIR" ;;
+        2) printf '%s' "$WORKER2_CACHE_DIR" ;;
+        3) printf '%s' "$WORKER3_CACHE_DIR" ;;
+    esac
+}
+_tp4_rank_vllm() {
+    case "$1" in
+        1) printf '%s' "$WORKER_VLLM_CACHE" ;;
+        2) printf '%s' "$WORKER2_VLLM_CACHE" ;;
+        3) printf '%s' "$WORKER3_VLLM_CACHE" ;;
+    esac
+}
+_tp4_rank_triton() {
+    case "$1" in
+        1) printf '%s' "$WORKER_TRITON_CACHE" ;;
+        2) printf '%s' "$WORKER2_TRITON_CACHE" ;;
+        3) printf '%s' "$WORKER3_TRITON_CACHE" ;;
+    esac
+}
+_tp4_rank_tilelang() {
+    case "$1" in
+        1) printf '%s' "$WORKER_TILELANG_CACHE" ;;
+        2) printf '%s' "$WORKER2_TILELANG_CACHE" ;;
+        3) printf '%s' "$WORKER3_TILELANG_CACHE" ;;
+    esac
+}
+_tp4_rank_cx7_if() {
+    case "$1" in
+        1) printf '%s' "$WORKER_CX7_IF" ;;
+        2) printf '%s' "$WORKER2_CX7_IF" ;;
+        3) printf '%s' "$WORKER3_CX7_IF" ;;
+    esac
+}
+_tp4_rank_cx7_ib() {
+    case "$1" in
+        1) printf '%s' "$WORKER_CX7_IB" ;;
+        2) printf '%s' "$WORKER2_CX7_IB" ;;
+        3) printf '%s' "$WORKER3_CX7_IB" ;;
+    esac
+}
+_tp4_rank_gid() {
+    case "$1" in
+        1) printf '%s' "$WORKER_GID" ;;
+        2) printf '%s' "$WORKER2_GID" ;;
+        3) printf '%s' "$WORKER3_GID" ;;
+    esac
+}
+_tp4_rank_container() {
+    case "$1" in
+        1) printf '%s' "$CONTAINER_WORKER" ;;
+        2) printf '%s' "$CONTAINER_WORKER2" ;;
+        3) printf '%s' "$CONTAINER_WORKER3" ;;
+    esac
+}
+_tp4_rank_nccl_dir() {
+    case "$1" in
+        1) printf '%s' "$WORKER_NCCL_HOST_DIR" ;;
+        2) printf '%s' "$WORKER2_NCCL_HOST_DIR" ;;
+        3) printf '%s' "$WORKER3_NCCL_HOST_DIR" ;;
+    esac
+}
+_tp4_first_ib() { printf '%s' "${1%%,*}"; }
+
+worker_ssh() { ssh -T -o BatchMode=yes -o ConnectTimeout=15 "$WORKER_SSH" "$@"; }
+worker_ssh_n() {
+    local r="$1"; shift
+    ssh -T -o BatchMode=yes -o ConnectTimeout=15 "$(_tp4_ssh_target "$r")" "$@"
+}
+
+usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
+
+count_shards() {
+    find "$1/snapshots" -name '*.safetensors' 2>/dev/null | wc -l | tr -d '[:space:]' || true
+}
+
+ensure_refs_main() {
+    local ref="$MODEL_PATH/refs/main" snap
+    [ -f "$ref" ] && [ -n "$(<"$ref")" ] && return 0
+    snap="$(ls -1t "$MODEL_PATH/snapshots" 2>/dev/null | head -n 1 || true)"
+    [ -n "$snap" ] || die "no snapshots under $MODEL_PATH — re-run download"
+    mkdir -p "$MODEL_PATH/refs"
+    printf '%s' "$snap" >"$ref"
+    log "wrote refs/main -> $snap (hf download left it empty)"
+}
+
+resolve_model_dir() {
+    local ref="$MODEL_PATH/refs/main" hash dir
+    ensure_refs_main
+    hash="$(<"$ref")"
+    dir="$MODEL_PATH/snapshots/$hash"
+    [ -f "$dir/config.json" ] || die "config.json missing in $dir — re-run with REFRESH_WEIGHTS=1"
+    printf '/root/.cache/huggingface/hub/%s/snapshots/%s' "$MODEL_CACHE_NAME" "$hash"
+}
+
+ensure_dflash_refs_main() {
+    local ref="$DFLASH_PATH/refs/main" snap
+    [ -f "$ref" ] && [ -n "$(<"$ref")" ] && return 0
+    snap="$(ls -1t "$DFLASH_PATH/snapshots" 2>/dev/null | head -n 1 || true)"
+    [ -n "$snap" ] || die "no snapshots under $DFLASH_PATH — re-run download"
+    mkdir -p "$DFLASH_PATH/refs"
+    printf '%s' "$snap" >"$ref"
+    log "wrote DFlash2 refs/main -> $snap"
+}
+
+resolve_dflash_dir() {
+    local ref="$DFLASH_PATH/refs/main" hash dir
+    ensure_dflash_refs_main
+    hash="$(<"$ref")"
+    dir="$DFLASH_PATH/snapshots/$hash"
+    [ -f "$dir/config.json" ] || die "DFlash2 config.json missing in $dir"
+    printf '/root/.cache/huggingface/hub/%s/snapshots/%s' "$DFLASH_CACHE_NAME" "$hash"
+}
+
+check_port_free() {
+    local port="$1" envname="$2"
+    command -v ss >/dev/null 2>&1 || return 0
+    if ss -ltn 2>/dev/null | awk '{print $4}' | grep -qE "[:.]${port}\$"; then
+        if docker inspect -f '{{.State.Running}}' "$CONTAINER_HEAD" 2>/dev/null | grep -q true; then
+            die "port ${port} is held by ${CONTAINER_HEAD} — use './start-tp4.sh restart' or './start-tp4.sh stop' first"
+        fi
+        die "port ${port} is already in use — stop it or rerun with ${envname}=<free-port>"
+    fi
+}
+
+trap 'warn "interrupted — containers keep running ('"'"'./start-tp4.sh logs'"'"' to watch, '"'"'./start-tp4.sh stop'"'"' to stop)"; exit 130' INT
+
+# ------------------------------ preflight ----------------------------------
+preflight() {
+    command -v docker  >/dev/null 2>&1 || die "docker not found on head"
+    command -v curl    >/dev/null 2>&1 || die "curl not found on head"
+    command -v rsync   >/dev/null 2>&1 || die "rsync not found on head"
+    docker info >/dev/null 2>&1 || die "cannot talk to docker daemon on head"
+
+    ip -4 addr show 2>/dev/null | grep -q "inet ${HEAD_IP}/" \
+        || die "HEAD_IP=${HEAD_IP} is not assigned on this host — set it in .env"
+
+    [ "$TP" = "4" ] || warn "TP=${TP} — this script is meant for TP=4"
+    [ "$NNODES" = "4" ] || warn "NNODES=${NNODES} — this script is meant for nnodes=4"
+    local r ssh_t
+    for r in 1 2 3; do
+        ssh_t="$(_tp4_ssh_target "$r")"
+        log "checking worker rank ${r} ${ssh_t} ..."
+        worker_ssh_n "$r" true 2>/dev/null \
+            || die "cannot ssh (key-based) to ${ssh_t} — set up passwordless ssh first"
+        worker_ssh_n "$r" "docker info >/dev/null 2>&1" \
+            || die "rank ${r} (${ssh_t}) cannot talk to its docker daemon"
+        worker_ssh_n "$r" "nvidia-smi -L 2>/dev/null | grep -q GB10" \
+            || warn "no GB10 GPU visible on rank ${r} (${ssh_t})"
+    done
+
+    # Each rank's GID index must name a populated entry on ITS OWN CX7 device.
+    # An empty (all-zero) entry passes every earlier check and then kills that
+    # rank ~60 s in with ibv_modify_qp errno 61 "No data available". The index is
+    # per-NIC, so validate head and worker separately: some pairs share one good
+    # index, others need different ones (HEAD_GID / WORKER_GID).
+    local gid_head gid_worker gid_path ib r ssh_t gid_ok=1
+    ib="$(_tp4_first_ib "$HEAD_CX7_IB")"
+    gid_path="/sys/class/infiniband/${ib}/ports/1/gids/${HEAD_GID}"
+    gid_head=$(cat "$gid_path" 2>/dev/null | tr -d ':0' || true)
+    [ -n "$gid_head" ] || { warn "head GID index ${HEAD_GID} is EMPTY on ${ib}"; gid_ok=0; }
+    for r in 1 2 3; do
+        ib="$(_tp4_first_ib "$(_tp4_rank_cx7_ib "$r")")"
+        gid_path="/sys/class/infiniband/${ib}/ports/1/gids/$(_tp4_rank_gid "$r")"
+        gid_worker=$(worker_ssh_n "$r" "cat '$gid_path' 2>/dev/null" | tr -d ':0' || true)
+        if [ -z "$gid_worker" ]; then
+            warn "rank ${r} GID index $(_tp4_rank_gid "$r") is EMPTY on ${ib}"
+            gid_ok=0
+        fi
+    done
+    if [ "$gid_ok" != "1" ]; then
+        if [ -z "$gid_head" ]; then
+            warn "head GID index ${HEAD_GID} is EMPTY on $(_tp4_first_ib "$HEAD_CX7_IB")"
+        fi
+        warn "GID tables — pick each node's ::ffff:<ip> entry whose type is RoCE v2;"
+        warn "the two indices need not match, and a v1 entry at the same index will not work:"
+        for i in 0 1 2 3 4 5 6 7; do
+            printf '    head   gid%s: %-40s %s\n' "$i" \
+                "$(cat "/sys/class/infiniband/${HEAD_CX7_IB}/ports/1/gids/$i" 2>/dev/null)" \
+                "$(cat "/sys/class/infiniband/${HEAD_CX7_IB}/ports/1/gid_attrs/types/$i" 2>/dev/null)" >&2
+        done
+        worker_ssh "for i in 0 1 2 3 4 5 6 7; do printf '    worker gid%s: %-40s %s\n' \"\$i\" \"\$(cat /sys/class/infiniband/${WORKER_CX7_IB}/ports/1/gids/\$i 2>/dev/null)\" \"\$(cat /sys/class/infiniband/${WORKER_CX7_IB}/ports/1/gid_attrs/types/\$i 2>/dev/null)\"; done" >&2 || true
+        die "set NCCL_IB_GID_INDEX (same index both ranks) or HEAD_GID/WORKER_GID (per rank) in .env to populated indices"
+    fi
+
+    [ "$TP" = "4" ] || warn "TP=${TP} — expected TP=4 for start-tp4.sh"
+    [ "$NNODES" = "4" ] || warn "NNODES=${NNODES} — expected 4"
+
+    local others cname
+    for r in 1 2 3; do
+        cname="$(_tp4_rank_container "$r")"
+        others=$(worker_ssh_n "$r" "docker ps --format '  {{.Names}}  ({{.Image}})'" 2>/dev/null | grep -v "^  ${cname}" || true)
+        if [ -n "$others" ]; then
+            warn "other containers are running on rank ${r} ($(_tp4_ssh_target "$r")):"
+            echo "$others" >&2
+            warn "this model needs most of each GB10 — stop GPU containers on that Spark first"
+        fi
+    done
+
+    check_port_free "$PORT" PORT
+    check_port_free "$MASTER_PORT" MASTER_PORT
+
+    [ -f "$STOP_PATCH_HOST" ] || die "$STOP_PATCH_HOST missing"
+    [ -f "$SCHED_PATCH_HOST" ] || die "$SCHED_PATCH_HOST missing"
+    [ -f "$DRAFTER_PATCH_HOST" ] || die "$DRAFTER_PATCH_HOST missing"
+    [ -f "$APC_PATCH_HOST" ] || die "$APC_PATCH_HOST missing"
+    [ -f "$XGRAMMAR_PATCH_HOST" ] || die "$XGRAMMAR_PATCH_HOST missing"
+    [ -f "$KPOOL_TAIL_PATCH_HOST" ] || die "$KPOOL_TAIL_PATCH_HOST missing"
+    [ -f "$SPINWAIT_PATCH_HOST" ] || die "$SPINWAIT_PATCH_HOST missing"
+    [ -f "$SCRIPT_DIR/overlay/patch_ablit.py" ] || die "$SCRIPT_DIR/overlay/patch_ablit.py missing"
+    [ -f "$SCRIPT_DIR/overlay/ablit_runtime.py" ] || die "$SCRIPT_DIR/overlay/ablit_runtime.py missing"
+    [ -f "$SCRIPT_DIR/ablit/LAYER_MAP.json" ] || die "$SCRIPT_DIR/ablit/LAYER_MAP.json missing"
+    if [ "$ABLIT" = "1" ]; then
+        log "ablit: ON (method=${ABLIT_METHOD} direction=${ABLIT_DIRECTION} layers=${ABLIT_LAYERS} alpha=${ABLIT_ALPHA} mtp=${ABLIT_INCLUDE_MTP})"
+    fi
+
+    local need_kb=$((180 * 1024 * 1024)) avail
+    mkdir -p "$HF_CACHE_DIR"
+    avail=$(df -Pk "$HF_CACHE_DIR" 2>/dev/null | awk 'NR==2{print $4}' || true)
+    [ "${avail:-0}" -ge "$need_kb" ] || warn "only $((avail/1024/1024)) GiB free on head for a ~164 GiB model"
+    for r in 1 2 3; do
+        avail=$(worker_ssh_n "$r" "df -Pk '$(_tp4_rank_home "$r")' 2>/dev/null" | awk 'NR==2{print $4}' || true)
+        [ "${avail:-0}" -ge "$need_kb" ] || warn "only $((avail/1024/1024)) GiB free on rank ${r} for a ~164 GiB model"
+    done
+
+    # The worker HF cache must be writable by the SSH user before the ~164 GiB
+    # sync starts. A root-owned ~/.cache/huggingface (prior sudo/docker
+    # prepare on the worker) otherwise fails mid-sync with a bare mkdir
+    # permission error. mkdir -p is idempotent and is what sync does anyway.
+    for r in 1 2 3; do
+        if ! worker_ssh_n "$r" "mkdir -p '$(_tp4_rank_hf "$r")/hub' && test -w '$(_tp4_rank_hf "$r")/hub'"; then
+            die "rank ${r} cannot write $(_tp4_rank_hf "$r")/hub — fix ownership, e.g. ssh $(_tp4_ssh_target "$r") \"sudo chown -R \$USER: '$(_tp4_rank_hf "$r")'\""
+        fi
+    done
+
+    log "preflight OK (head=$(hostname) ${HEAD_IP}, workers=${WORKER_SSH} ${WORKER2_SSH} ${WORKER3_SSH})"
+}
+
+# ------------------------------ image --------------------------------------
+image_from_registry() {
+    case "$IMAGE" in
+        */*) return 0 ;;
+        *) return 1 ;;
+    esac
+}
+
+login_ghcr_if_token() {
+    [ -n "${GHCR_TOKEN:-}" ] || return 0
+    log "docker login ghcr.io as ${GHCR_USER} (GHCR_TOKEN)"
+    echo "$GHCR_TOKEN" | docker login ghcr.io -u "$GHCR_USER" --password-stdin >/dev/null
+}
+
+login_ghcr_if_token_worker() {
+    [ -n "${GHCR_TOKEN:-}" ] || return 0
+    local r
+    for r in 1 2 3; do
+        log "docker login ghcr.io on rank ${r} as ${GHCR_USER} (GHCR_TOKEN)"
+        echo "$GHCR_TOKEN" | worker_ssh_n "$r" "docker login ghcr.io -u '$GHCR_USER' --password-stdin" >/dev/null
+    done
+}
+
+# RepoDigest is stable across overlay2 vs containerd. Those snapshotters
+# disagree on .Id (config digest vs index digest), so start.sh used to
+# ship even after the worker had already pulled the same GHCR tag (issue #8).
+# Local builds have no RepoDigest — use RootFS layer diffs, then .Id.
+_IMAGE_KEY_FMT='{{if .RepoDigests}}{{index .RepoDigests 0}}{{else if .RootFS.Layers}}{{join .RootFS.Layers ","}}{{else}}{{.Id}}{{end}}'
+
+parse_image_key() {
+    tr -d '\r' | sed -n 's/^GLM53KEY //p' | tail -n 1
+}
+
+local_image_key() {
+    docker image inspect -f "GLM53KEY ${_IMAGE_KEY_FMT}" "$IMAGE" 2>/dev/null | parse_image_key
+}
+
+worker_image_key() {
+    worker_ssh_n "${1:-1}" "docker image inspect -f 'GLM53KEY ${_IMAGE_KEY_FMT}' '$IMAGE' 2>/dev/null" | parse_image_key
+}
+
+images_match() {
+    [ -n "${1:-}" ] && [ -n "${2:-}" ] && [ "$1" = "$2" ]
+}
+
+image_platform() {
+    if [ -n "${IMAGE_PLATFORM:-}" ]; then
+        printf '%s' "$IMAGE_PLATFORM"
+        return
+    fi
+    local p
+    p="$(docker image inspect -f '{{.Os}}/{{.Architecture}}' "$IMAGE" 2>/dev/null || true)"
+    printf '%s' "${p:-linux/arm64}"
+}
+
+# Hash of Dockerfile + overlay/tests/files/ablit inputs that docker COPY.
+# Compared to LABEL glm53.recipe.stamp so a git pull rebuilds once.
+overlay_recipe_hash() {
+    {
+        printf '%s\n' "$SCRIPT_DIR/Dockerfile"
+        find "$SCRIPT_DIR/overlay" "$SCRIPT_DIR/files" "$SCRIPT_DIR/tests" \
+            "$SCRIPT_DIR/ablit" \
+            -type f \
+            ! -path '*/__pycache__/*' \
+            ! -path '*/ablit/transplant/*' \
+            ! -name '*.pyc' \
+            2>/dev/null
+    } | LC_ALL=C sort | xargs -d '\n' -r sha256sum | sha256sum | awk '{print $1}'
+}
+
+image_recipe_stamp() {
+    local stamp
+    stamp="$(docker image inspect -f '{{ index .Config.Labels "glm53.recipe.stamp" }}' "$IMAGE" 2>/dev/null || true)"
+    case "$stamp" in
+        ""|"<no value>"|"<nil>") printf '' ;;
+        *) printf '%s' "$stamp" ;;
+    esac
+}
+
+build_image() {
+    local stamp
+    stamp="$(overlay_recipe_hash)"
+    log "building ${IMAGE} from Dockerfile stamp=${stamp:0:12} (log: $LOGDIR/build-sm121.log) ..."
+    docker build --build-arg "GLM53_RECIPE_STAMP=$stamp" -t "$IMAGE" "$SCRIPT_DIR" \
+        >"$LOGDIR/build-sm121.log" 2>&1 \
+        || { tail -n 40 "$LOGDIR/build-sm121.log" >&2; die "docker build of $IMAGE failed"; }
+}
+
+pull_image() {
+    login_ghcr_if_token
+    log "pulling ${IMAGE} ..."
+    docker pull "$IMAGE" && return 0
+    die "docker pull ${IMAGE} failed.
+  :exl3 is a public GHCR package — check network / disk.
+  If you still get 401/403: echo YOUR_PAT | docker login ghcr.io -u YOUR_GITHUB_USER --password-stdin
+  Overlay rebuild: BUILD=1 ./start.sh. Recipe-stamp drift also rebuilds; SKIP_BUILD=1 keeps GHCR."
+}
+
+pull_image_on_worker() {
+    local r="${1:-1}"
+    login_ghcr_if_token_worker
+    log "pulling ${IMAGE} on rank ${r} ..."
+    worker_ssh_n "$r" "docker pull '$IMAGE'"
+}
+
+ship_image_to_worker() {
+    local r="${1:-1}" platform
+    platform="$(image_platform)"
+    log "shipping ${IMAGE} (${platform}) to rank ${r} via docker save | ssh docker load ..."
+    if docker save --platform "$platform" "$IMAGE" | worker_ssh_n "$r" docker load; then
+        return 0
+    fi
+    warn "docker save --platform ${platform} failed — retrying without --platform"
+    docker save "$IMAGE" | worker_ssh_n "$r" docker load
+}
+
+ensure_image() {
+    mkdir -p "$LOGDIR"
+    local head_ok=0 worker_ok=0 head_key="" worker_key=""
+    if docker image inspect "$IMAGE" >/dev/null 2>&1; then
+        head_ok=1
+        head_key="$(local_image_key)"
+    fi
+    worker_ok=1
+    for r in 1 2 3; do
+        if worker_ssh_n "$r" "docker image inspect '$IMAGE' >/dev/null 2>&1"; then
+            worker_key="$(worker_image_key "$r")"
+            if images_match "$head_key" "$worker_key"; then
+                :
+            else
+                worker_ok=0
+                log "rank ${r} image differs (head=${head_key:-none} worker=${worker_key:-none}) — will refresh"
+            fi
+        else
+            worker_ok=0
+        fi
+    done
+    local skip_pull="${SKIP_PULL:-0}"
+    [ "${PULL:-0}" = "1" ] && skip_pull=0
+    local wanted_stamp have_stamp
+    wanted_stamp="$(overlay_recipe_hash)"
+    have_stamp=""
+    [ "$head_ok" = "1" ] && have_stamp="$(image_recipe_stamp)"
+    if [ "${BUILD:-0}" != "1" ] && [ "${SKIP_BUILD:-0}" != "1" ]; then
+        if [ "$head_ok" = "0" ] || [ "$have_stamp" != "$wanted_stamp" ]; then
+            log "image recipe ${have_stamp:-none} != repo ${wanted_stamp:0:12} — rebuilding (SKIP_BUILD=1 keeps GHCR)"
+            BUILD=1
+        fi
+    elif [ "${SKIP_BUILD:-0}" = "1" ] && [ "$have_stamp" != "$wanted_stamp" ]; then
+        warn "SKIP_BUILD=1 — not rebuilding; stamp ${have_stamp:-none} != repo ${wanted_stamp:0:12}"
+    fi
+    if [ "${BUILD:-0}" = "1" ]; then
+        build_image
+        head_key="$(local_image_key)"
+        head_ok=1
+        worker_ok=0
+    elif image_from_registry && [ "$skip_pull" != "1" ]; then
+        local before_key="$head_key"
+        pull_image
+        head_key="$(local_image_key)"
+        head_ok=1
+        if [ "$head_key" != "$before_key" ]; then
+            log "pulled ${IMAGE} (${before_key:-missing} -> ${head_key})"
+        else
+            log "${IMAGE} already current"
+        fi
+        if images_match "$worker_key" "$head_key"; then
+            worker_ok=1
+        else
+            worker_ok=0
+        fi
+    elif [ "$head_ok" = "0" ]; then
+        if image_from_registry && [ "$skip_pull" = "1" ]; then
+            die "SKIP_PULL=1 but ${IMAGE} is not on the head"
+        fi
+        build_image
+        head_key="$(local_image_key)"
+        head_ok=1
+        worker_ok=0
+    fi
+    if [ "${SKIP_SHIP:-0}" = "1" ]; then
+        [ "$worker_ok" = "1" ] || warn "SKIP_SHIP=1 — not copying ${IMAGE} to workers"
+    elif [ "$worker_ok" = "0" ]; then
+        for r in 1 2 3; do
+            local wok=0
+            worker_key="$(worker_image_key "$r")"
+            if images_match "$head_key" "$worker_key"; then
+                continue
+            fi
+            if image_from_registry && [ "$skip_pull" != "1" ] && [ "${BUILD:-0}" != "1" ]; then
+                if pull_image_on_worker "$r"; then
+                    worker_key="$(worker_image_key "$r")"
+                    if images_match "$head_key" "$worker_key"; then
+                        wok=1
+                        log "rank ${r} pulled ${IMAGE} — matches head"
+                    else
+                        warn "rank ${r} pull left a different image — shipping"
+                    fi
+                else
+                    warn "rank ${r} docker pull failed — shipping over SSH"
+                fi
+            fi
+            if [ "$wok" = "0" ]; then
+                ship_image_to_worker "$r"
+                worker_key="$(worker_image_key "$r")"
+                if images_match "$head_key" "$worker_key"; then
+                    :
+                elif worker_ssh_n "$r" "docker image inspect '$IMAGE' >/dev/null 2>&1"; then
+                    warn "rank ${r} has ${IMAGE} after ship but keys still differ — continuing"
+                else
+                    die "rank ${r} still missing ${IMAGE} after ship"
+                fi
+            fi
+        done
+    fi
+    if [ "${SKIP_OVERLAY_VERIFY:-0}" != "1" ]; then
+        log "GPU EXL3 self-check on ${IMAGE} (log: $LOGDIR/overlay-verify.log) ..."
+        docker run --rm --gpus all \
+            -e EXL3_SELFCHECK_GPU=1 \
+            --entrypoint python3 "$IMAGE" /opt/glm53/test_exl3_overlay.py \
+            >"$LOGDIR/overlay-verify.log" 2>&1 \
+            || { tail -n 80 "$LOGDIR/overlay-verify.log" >&2; die "EXL3 overlay GPU self-check failed"; }
+        log "overlay verify OK"
+    fi
+    log "image ready on all four nodes"
+}
+
+# ---------------------------- weight download ------------------------------
+# Use an already-complete local tree (primary or upstream fallback). If the
+# durable Mia-AiLab mirror is still filling / 404s, keep serving from the
+# brandonmusic cache folder without a second 164 GiB pull.
+adopt_complete_weights() {
+    local have
+    have="$(count_shards "$MODEL_PATH")"
+    if [ "${have:-0}" -ge "$EXPECTED_SHARDS" ]; then
+        ensure_refs_main
+        log "weights already present: $MODEL_PATH ($have shards)"
+        return 0
+    fi
+    have="$(count_shards "$FALLBACK_MODEL_PATH")"
+    if [ "${have:-0}" -ge "$EXPECTED_SHARDS" ]; then
+        log "primary cache incomplete — using fallback ${MODEL_FALLBACK} at $FALLBACK_MODEL_PATH ($have shards)"
+        MODEL_PATH="$FALLBACK_MODEL_PATH"
+        MODEL_CACHE_NAME="$MODEL_FALLBACK_CACHE_NAME"
+        ensure_refs_main
+        return 0
+    fi
+    return 1
+}
+
+# Resolve the HF CLI even when it lives outside PATH (venv installs), with a
+# python huggingface_hub fallback when no binary exists (issue #22, item 1).
+# Sets the global HF_BIN_CMD array. HF_BIN (may contain arguments) wins when
+# its first word resolves. Returns 1 when nothing usable is found.
+resolve_hf_bin() {
+    HF_BIN_CMD=()
+    if [ -n "${HF_BIN:-}" ]; then
+        read -ra HF_BIN_CMD <<< "$HF_BIN"
+        if command -v "${HF_BIN_CMD[0]}" >/dev/null 2>&1; then return 0; fi
+        HF_BIN_CMD=()
+    fi
+    local cand
+    for cand in hf huggingface-cli "$HOME/.local/bin/hf" "$HOME/.hf-cli/venv/bin/hf" /opt/hf-cli/venv/bin/hf; do
+        if command -v "$cand" >/dev/null 2>&1; then HF_BIN_CMD=("$cand"); return 0; fi
+    done
+    if command -v python3 >/dev/null 2>&1 && python3 -c 'import huggingface_hub' >/dev/null 2>&1; then
+        HF_BIN_CMD=(python3 -m huggingface_hub.commands.huggingface_cli)
+        return 0
+    fi
+    return 1
+}
+
+hf_download_repo() {
+    local repo="$1"
+    shift
+    local -a args=("$repo")
+    if [ -n "${MODEL_REVISION:-}" ] && [ "$repo" = "$MODEL" ]; then
+        args+=(--revision "$MODEL_REVISION")
+    fi
+    args+=("$@")
+    HF_HOME="$HF_CACHE_DIR" "${HF_BIN_CMD[@]}" download "${args[@]}"
+}
+
+download_weights() {
+    [ "${SKIP_DOWNLOAD:-0}" = "1" ] && { log "SKIP_DOWNLOAD=1 — skipping download check"; return; }
+    if [ "${REFRESH_WEIGHTS:-0}" != "1" ] && adopt_complete_weights; then
+        return
+    fi
+
+    resolve_hf_bin || die "no 'hf' / 'huggingface-cli' on PATH and no python huggingface_hub — pip install --user -U 'huggingface_hub[cli]' (or set HF_BIN=/path/to/hf)"
+
+    mkdir -p "$HF_CACHE_DIR"
+    local -a hf_excl=()
+    local pat
+    IFS=',' read -ra _excl_pats <<< "${HF_DOWNLOAD_EXCLUDE:-runtime-results/**,src/**,runtime/src/**,scripts/**,docs/**,results/**,.materialization/**,runtime/scripts/**}"
+    for pat in "${_excl_pats[@]}"; do
+        [ -n "$pat" ] && hf_excl+=(--exclude "$pat")
+    done
+
+    log "downloading ${MODEL} (~164 GiB / ${EXPECTED_SHARDS} shards) into ${HF_CACHE_DIR} ..."
+    hf_download_repo "$MODEL" "${hf_excl[@]}" || warn "download of ${MODEL} failed — will try ${MODEL_FALLBACK}"
+    if adopt_complete_weights; then
+        return
+    fi
+
+    if [ "$MODEL_FALLBACK" != "$MODEL" ]; then
+        log "falling back to ${MODEL_FALLBACK} ..."
+        hf_download_repo "$MODEL_FALLBACK" "${hf_excl[@]}" \
+            || die "download of ${MODEL} and ${MODEL_FALLBACK} both failed"
+    fi
+    adopt_complete_weights \
+        || die "download finished with $(count_shards "$MODEL_PATH") / $EXPECTED_SHARDS shards"
+}
+
+download_dflash() {
+    [ "$SPEC_METHOD" = "dflash" ] || return 0
+    [ "${SKIP_DOWNLOAD:-0}" = "1" ] && { log "SKIP_DOWNLOAD=1 — skipping DFlash2 download check"; return; }
+    local have
+    have="$(find "$DFLASH_PATH/snapshots" -name 'model.safetensors' 2>/dev/null | wc -l | tr -d '[:space:]' || true)"
+    if [ "${have:-0}" -ge 1 ] && [ "${REFRESH_WEIGHTS:-0}" != "1" ]; then
+        log "DFlash2 already present: $DFLASH_PATH"
+        ensure_dflash_refs_main
+        return
+    fi
+    resolve_hf_bin || die "no 'hf' / 'huggingface-cli' on PATH and no python huggingface_hub — pip install --user -U 'huggingface_hub[cli]' (or set HF_BIN=/path/to/hf)"
+    mkdir -p "$HF_CACHE_DIR"
+    log "downloading ${DFLASH_MODEL} (~2.3 GiB) into ${HF_CACHE_DIR} ..."
+    HF_HOME="$HF_CACHE_DIR" "${HF_BIN_CMD[@]}" download "$DFLASH_MODEL"
+    ensure_dflash_refs_main
+    have="$(find "$DFLASH_PATH/snapshots" -name 'model.safetensors' 2>/dev/null | wc -l | tr -d '[:space:]' || true)"
+    [ "${have:-0}" -ge 1 ] || die "DFlash2 download finished without model.safetensors"
+    log "DFlash2 download complete"
+}
+
+# Head-only Hub fetch. No docker, no SSH, no worker rsync.
+download_only() {
+    local have
+    resolve_hf_bin || die "no 'hf' / 'huggingface-cli' on PATH and no python huggingface_hub — pip install --user -U 'huggingface_hub[cli]' (or set HF_BIN=/path/to/hf)"
+    mkdir -p "$HF_CACHE_DIR"
+    local need_kb=$((180 * 1024 * 1024)) avail
+    avail=$(df -Pk "$HF_CACHE_DIR" 2>/dev/null | awk 'NR==2{print $4}' || true)
+    [ "${avail:-0}" -ge "$need_kb" ] || warn "only $((avail/1024/1024)) GiB free on this disk for a ~164 GiB model"
+
+    # Explicit download: do not honor SKIP_DOWNLOAD from .env.
+    SKIP_DOWNLOAD=0
+    download_weights
+    download_dflash
+
+    have="$(count_shards "$MODEL_PATH")"
+    log "======================================================================"
+    log "head HF cache : ${HF_CACHE_DIR}"
+    log "  target      : ${MODEL}  (${have} / ${EXPECTED_SHARDS} shards)"
+    log "  snapshot    : ${MODEL_PATH}"
+    if [ "$SPEC_METHOD" = "dflash" ]; then
+        log "  DFlash2     : ${DFLASH_MODEL}"
+        log "  draft cache : ${DFLASH_PATH}"
+    else
+        log "  DFlash2     : skipped (SPEC_METHOD=${SPEC_METHOD})"
+    fi
+    log "workers were not touched. ./start-tp4.sh will rsync on launch unless SKIP_SYNC=1."
+    log "======================================================================"
+}
+
+# ------------------------------ weight sync --------------------------------
+# Keyed on the snapshot commit (refs/main, with the same repair fallback as
+# ensure_refs_main), not on MODEL_REVISION: the marker lives inside each
+# synced repo folder, so a MODEL / revision switch re-syncs automatically.
+# Without it, every ./start.sh pays a full size+mtime re-verification walk
+# over ~164 GiB / 120 shards on both ends for zero bytes of difference
+# (issue #22, item 2). FORCE_SYNC=1 bypasses the marker; deleting the
+# marker file on the worker has the same effect.
+sync_repo_marker_rev() {
+    local src="$1"
+    local rev
+    rev="$(cat "$src/refs/main" 2>/dev/null || true)"
+    [ -n "$rev" ] || rev="$(ls -1t "$src/snapshots" 2>/dev/null | head -n 1 || true)"
+    [ -n "$rev" ] || rev="unknown"
+    printf '%s' "$rev"
+}
+
+sync_repo_to_one_worker() {
+    local r="$1" src="$2" cache_name="$3" label="$4"
+    local marker rev hf ssh_t
+    hf="$(_tp4_rank_hf "$r")"
+    ssh_t="$(_tp4_ssh_target "$r")"
+    marker="${hf}/hub/${cache_name}/.glm53-exl3-synced"
+    rev="$(sync_repo_marker_rev "$src")"
+    if [ "${FORCE_SYNC:-0}" != "1" ] \
+       && [ "$(worker_ssh_n "$r" "cat '$marker' 2>/dev/null" || true)" = "$rev" ]; then
+        log "rank ${r} ${cache_name} already at ${rev} — rsync skipped"
+        return 0
+    fi
+    log "syncing ${label} to rank ${r} (${ssh_t}) ..."
+    worker_ssh_n "$r" "mkdir -p '${hf}/hub/${cache_name}'"
+    rsync -a --partial --info=progress2 \
+        "$src/" "${ssh_t}:${hf}/hub/${cache_name}/"
+    worker_ssh_n "$r" "printf '%s' '$rev' > '$marker'"
+}
+
+sync_weights() {
+    [ "${SKIP_SYNC:-0}" = "1" ] && { log "SKIP_SYNC=1 — not syncing to workers"; return; }
+    [ -d "$MODEL_PATH" ] || die "weights missing at $MODEL_PATH — run without SKIP_DOWNLOAD first"
+    local r
+    for r in 1 2 3; do
+        sync_repo_to_one_worker "$r" "$MODEL_PATH" "$MODEL_CACHE_NAME" "weights"
+        if [ "$SPEC_METHOD" = "dflash" ]; then
+            [ -d "$DFLASH_PATH" ] || die "DFlash2 weights missing at $DFLASH_PATH"
+            sync_repo_to_one_worker "$r" "$DFLASH_PATH" "$DFLASH_CACHE_NAME" "DFlash2 draft"
+        fi
+    done
+    log "all worker weights in sync"
+}
+
+# ------------------------ inner container scripts --------------------------
+write_inner_scripts() {
+    cat > "$HEAD_SCRIPT" <<'EOF'
+#!/bin/bash
+set -euo pipefail
+say() { echo "[glm53-exl3-head] $*"; }
+
+ARGS=(
+    --served-model-name "${SERVED_MODEL_NAME}"
+    --host 0.0.0.0
+    --port "${PORT}"
+    --tensor-parallel-size "${TP}"
+    --nnodes "${NNODES}"
+    --node-rank 0
+    --master-addr "${HEAD_IP}"
+    --master-port "${MASTER_PORT}"
+    --distributed-executor-backend mp
+    --tool-call-parser glm47
+    --enable-auto-tool-choice
+    --reasoning-parser glm45
+    --enable-prefix-caching
+    --no-enable-flashinfer-autotune
+)
+[ "${ENFORCE_EAGER:-1}" = "1" ] && ARGS+=(--enforce-eager)
+[ -n "${QUANTIZATION:-}" ] && [ "${QUANTIZATION}" != "none" ] && ARGS+=(--quantization "${QUANTIZATION}")
+[ -n "${MAX_MODEL_LEN:-}" ] && ARGS+=(--max-model-len "${MAX_MODEL_LEN}")
+[ -n "${GPU_MEM_UTIL:-}" ]  && ARGS+=(--gpu-memory-utilization "${GPU_MEM_UTIL}")
+[ -n "${MAX_NUM_SEQS:-}" ] && ARGS+=(--max-num-seqs "${MAX_NUM_SEQS}")
+[ -n "${MAX_NUM_BATCHED_TOKENS:-}" ] && ARGS+=(--max-num-batched-tokens "${MAX_NUM_BATCHED_TOKENS}")
+[ -n "${KV_CACHE_DTYPE:-}" ] && ARGS+=(--kv-cache-dtype "${KV_CACHE_DTYPE}")
+if [ "${SPEC_METHOD:-mtp}" = "dflash" ]; then
+    ARGS+=(--speculative-config "$(python3 -S -c 'import json,os
+spec={"method":"dflash","model":os.environ["DFLASH_MODEL_DIR"],"num_speculative_tokens":int(os.environ.get("DFLASH_TOKENS","7")),"kv_cache_dtype":"auto","draft_sample_method":"probabilistic","rejection_sample_method":"standard"}
+tp=os.environ.get("DFLASH_DRAFT_TP","").strip()
+if tp:
+    spec["draft_tensor_parallel_size"]=int(tp)
+print(json.dumps(spec,separators=(",",":")))')")
+elif [ "${SPEC_METHOD:-mtp}" = "none" ]; then
+    :
+elif [ "${MTP_TOKENS:-0}" != "0" ]; then
+    ARGS+=(--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":${MTP_TOKENS}}")
+fi
+if [ -n "${CHAT_TEMPLATE:-}" ] && [ -f "${CHAT_TEMPLATE}" ]; then
+    ARGS+=(--chat-template "${CHAT_TEMPLATE}")
+fi
+if [ "${LANGUAGE_MODEL_ONLY:-0}" = "1" ]; then
+    ARGS+=(--language-model-only)
+    say "language-model-only: no vision tower"
+else
+    [ -n "${LIMIT_MM:-}" ] && ARGS+=(--limit-mm-per-prompt "${LIMIT_MM}")
+    [ "${SKIP_MM_PROFILING:-1}" = "1" ] && ARGS+=(--skip-mm-profiling)
+    say "vision on: limit-mm=${LIMIT_MM:-} skip-mm-profiling=${SKIP_MM_PROFILING:-1} chat-template=${CHAT_TEMPLATE:-}"
+fi
+if [ -n "${EXTRA_ARGS:-}" ]; then
+    # shellcheck disable=SC2206
+    EXTRA=(${EXTRA_ARGS})
+    ARGS+=("${EXTRA[@]}")
+fi
+
+[ -f "${MODEL_DIR}/config.json" ] || { say "FATAL: ${MODEL_DIR}/config.json missing"; ls -la "${MODEL_DIR}" | head; exit 1; }
+if [ -f /opt/glm53/patch_glm_video_placeholders.py ]; then
+    python3 /opt/glm53/patch_glm_video_placeholders.py
+fi
+if [ -f /opt/glm53/patch_suppress_stops_in_reasoning.py ]; then
+    python3 /opt/glm53/patch_suppress_stops_in_reasoning.py
+fi
+if [ -f /opt/glm53/patch_scheduler_decode_floor.py ]; then
+    python3 /opt/glm53/patch_scheduler_decode_floor.py
+fi
+if [ -f /opt/glm53/patch_glm5_drafter_group.py ]; then
+    python3 /opt/glm53/patch_glm5_drafter_group.py
+fi
+if [ -f /opt/glm53/patch_hybrid_prefix_hit.py ]; then
+    python3 /opt/glm53/patch_hybrid_prefix_hit.py
+fi
+if [ -f /opt/glm53/patch_xgrammar_termination.py ]; then
+    python3 /opt/glm53/patch_xgrammar_termination.py
+fi
+if [ -f /opt/glm53/patch_kpool_tail_slotmap.py ]; then
+    python3 /opt/glm53/patch_kpool_tail_slotmap.py
+fi
+if [ -f /opt/glm53/patch_spinwait.py ]; then
+    python3 /opt/glm53/patch_spinwait.py
+fi
+if [ -f /opt/glm53/patch_indexer_workspace.py ]; then
+    python3 /opt/glm53/patch_indexer_workspace.py
+fi
+if [ -f /opt/glm53/patch_ablit.py ]; then
+    python3 /opt/glm53/patch_ablit.py
+fi
+if [ "${ABLIT:-0}" = "1" ]; then
+    say "ablit: o_proj orthogonalization ON (method=${ABLIT_METHOD:-auto} direction=${ABLIT_DIRECTION:-dealign} layers=${ABLIT_LAYERS:-15-45} alpha=${ABLIT_ALPHA:-3.0})"
+else
+    say "ablit: off — stock o_proj weights"
+fi
+say "launching: vllm serve ${MODEL_DIR} ${ARGS[*]}"
+exec vllm serve "${MODEL_DIR}" "${ARGS[@]}"
+EOF
+
+    cat > "$WORKER_SCRIPT" <<'EOF'
+#!/bin/bash
+set -euo pipefail
+say() { echo "[glm53-exl3-worker] $*"; }
+
+ARGS=(
+    --served-model-name "${SERVED_MODEL_NAME}"
+    --host 0.0.0.0
+    --port "${PORT}"
+    --tensor-parallel-size "${TP}"
+    --nnodes "${NNODES}"
+    --node-rank "${NODE_RANK}"
+    --master-addr "${HEAD_IP}"
+    --master-port "${MASTER_PORT}"
+    --distributed-executor-backend mp
+    --headless
+    --tool-call-parser glm47
+    --enable-auto-tool-choice
+    --reasoning-parser glm45
+    --enable-prefix-caching
+    --no-enable-flashinfer-autotune
+)
+[ "${ENFORCE_EAGER:-1}" = "1" ] && ARGS+=(--enforce-eager)
+[ -n "${QUANTIZATION:-}" ] && [ "${QUANTIZATION}" != "none" ] && ARGS+=(--quantization "${QUANTIZATION}")
+[ -n "${MAX_MODEL_LEN:-}" ] && ARGS+=(--max-model-len "${MAX_MODEL_LEN}")
+[ -n "${GPU_MEM_UTIL:-}" ]  && ARGS+=(--gpu-memory-utilization "${GPU_MEM_UTIL}")
+[ -n "${MAX_NUM_SEQS:-}" ] && ARGS+=(--max-num-seqs "${MAX_NUM_SEQS}")
+[ -n "${MAX_NUM_BATCHED_TOKENS:-}" ] && ARGS+=(--max-num-batched-tokens "${MAX_NUM_BATCHED_TOKENS}")
+[ -n "${KV_CACHE_DTYPE:-}" ] && ARGS+=(--kv-cache-dtype "${KV_CACHE_DTYPE}")
+if [ "${SPEC_METHOD:-mtp}" = "dflash" ]; then
+    ARGS+=(--speculative-config "$(python3 -S -c 'import json,os
+spec={"method":"dflash","model":os.environ["DFLASH_MODEL_DIR"],"num_speculative_tokens":int(os.environ.get("DFLASH_TOKENS","7")),"kv_cache_dtype":"auto","draft_sample_method":"probabilistic","rejection_sample_method":"standard"}
+tp=os.environ.get("DFLASH_DRAFT_TP","").strip()
+if tp:
+    spec["draft_tensor_parallel_size"]=int(tp)
+print(json.dumps(spec,separators=(",",":")))')")
+elif [ "${SPEC_METHOD:-mtp}" = "none" ]; then
+    :
+elif [ "${MTP_TOKENS:-0}" != "0" ]; then
+    ARGS+=(--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":${MTP_TOKENS}}")
+fi
+if [ -n "${CHAT_TEMPLATE:-}" ] && [ -f "${CHAT_TEMPLATE}" ]; then
+    ARGS+=(--chat-template "${CHAT_TEMPLATE}")
+fi
+if [ "${LANGUAGE_MODEL_ONLY:-0}" = "1" ]; then
+    ARGS+=(--language-model-only)
+else
+    [ -n "${LIMIT_MM:-}" ] && ARGS+=(--limit-mm-per-prompt "${LIMIT_MM}")
+    [ "${SKIP_MM_PROFILING:-1}" = "1" ] && ARGS+=(--skip-mm-profiling)
+fi
+if [ -n "${EXTRA_ARGS:-}" ]; then
+    # shellcheck disable=SC2206
+    EXTRA=(${EXTRA_ARGS})
+    ARGS+=("${EXTRA[@]}")
+fi
+
+[ -f "${MODEL_DIR}/config.json" ] || { say "FATAL: ${MODEL_DIR}/config.json missing"; ls -la "${MODEL_DIR}" | head; exit 1; }
+if [ -f /opt/glm53/patch_glm_video_placeholders.py ]; then
+    python3 /opt/glm53/patch_glm_video_placeholders.py
+fi
+if [ -f /opt/glm53/patch_suppress_stops_in_reasoning.py ]; then
+    python3 /opt/glm53/patch_suppress_stops_in_reasoning.py
+fi
+if [ -f /opt/glm53/patch_scheduler_decode_floor.py ]; then
+    python3 /opt/glm53/patch_scheduler_decode_floor.py
+fi
+if [ -f /opt/glm53/patch_glm5_drafter_group.py ]; then
+    python3 /opt/glm53/patch_glm5_drafter_group.py
+fi
+if [ -f /opt/glm53/patch_hybrid_prefix_hit.py ]; then
+    python3 /opt/glm53/patch_hybrid_prefix_hit.py
+fi
+if [ -f /opt/glm53/patch_xgrammar_termination.py ]; then
+    python3 /opt/glm53/patch_xgrammar_termination.py
+fi
+if [ -f /opt/glm53/patch_kpool_tail_slotmap.py ]; then
+    python3 /opt/glm53/patch_kpool_tail_slotmap.py
+fi
+if [ -f /opt/glm53/patch_spinwait.py ]; then
+    python3 /opt/glm53/patch_spinwait.py
+fi
+if [ -f /opt/glm53/patch_indexer_workspace.py ]; then
+    python3 /opt/glm53/patch_indexer_workspace.py
+fi
+if [ -f /opt/glm53/patch_ablit.py ]; then
+    python3 /opt/glm53/patch_ablit.py
+fi
+if [ "${ABLIT:-0}" = "1" ]; then
+    say "ablit: o_proj orthogonalization ON (method=${ABLIT_METHOD:-auto} direction=${ABLIT_DIRECTION:-dealign} layers=${ABLIT_LAYERS:-15-45} alpha=${ABLIT_ALPHA:-3.0})"
+else
+    say "ablit: off — stock o_proj weights"
+fi
+say "joining TP${TP} at ${HEAD_IP}:${MASTER_PORT} as rank ${NODE_RANK}"
+exec vllm serve "${MODEL_DIR}" "${ARGS[@]}"
+EOF
+    chmod +x "$HEAD_SCRIPT" "$WORKER_SCRIPT"
+}
+
+# ------------------------------- launch ------------------------------------
+_tp4_scp_runtime() {
+    local r="$1" ssh_t cname
+    ssh_t="$(_tp4_ssh_target "$r")"
+    cname="$(_tp4_rank_container "$r")"
+    scp -q -o BatchMode=yes "$WORKER_SCRIPT" "${ssh_t}:/tmp/${cname}.sh"
+    scp -q -o BatchMode=yes "$CHAT_TEMPLATE_HOST" "${ssh_t}:/tmp/glm53-chat_template.jinja"
+    scp -q -o BatchMode=yes "$VIDEO_PATCH_HOST" "${ssh_t}:/tmp/patch_glm_video_placeholders.py"
+    scp -q -o BatchMode=yes "$STOP_PATCH_HOST" "${ssh_t}:/tmp/patch_suppress_stops_in_reasoning.py"
+    scp -q -o BatchMode=yes "$SCHED_PATCH_HOST" "${ssh_t}:/tmp/patch_scheduler_decode_floor.py"
+    scp -q -o BatchMode=yes "$DRAFTER_PATCH_HOST" "${ssh_t}:/tmp/patch_glm5_drafter_group.py"
+    scp -q -o BatchMode=yes "$APC_PATCH_HOST" "${ssh_t}:/tmp/patch_hybrid_prefix_hit.py"
+    scp -q -o BatchMode=yes "$XGRAMMAR_PATCH_HOST" "${ssh_t}:/tmp/patch_xgrammar_termination.py"
+    scp -q -o BatchMode=yes "$KPOOL_TAIL_PATCH_HOST" "${ssh_t}:/tmp/patch_kpool_tail_slotmap.py"
+    scp -q -o BatchMode=yes "$SPINWAIT_PATCH_HOST" "${ssh_t}:/tmp/patch_spinwait.py"
+    worker_ssh_n "$r" "rm -rf /tmp/glm53-ablit"
+    scp -q -r -o BatchMode=yes "$SCRIPT_DIR/ablit" "${ssh_t}:/tmp/glm53-ablit"
+    scp -q -o BatchMode=yes "$SCRIPT_DIR/overlay/ablit_runtime.py" "${ssh_t}:/tmp/glm53-ablit_runtime.py"
+    scp -q -o BatchMode=yes "$SCRIPT_DIR/overlay/patch_ablit.py" "${ssh_t}:/tmp/patch_ablit.py"
+}
+
+launch_cluster() {
+    local r
+    docker rm -f "$CONTAINER_HEAD" >/dev/null 2>&1 || true
+    for r in 1 2 3; do
+        worker_ssh_n "$r" "docker rm -f '$(_tp4_rank_container "$r")'" >/dev/null 2>&1 || true
+    done
+
+    mkdir -p "$CACHE_ROOT" "$TRITON_HOST_CACHE" "$TILELANG_HOST_CACHE"
+    [ -f "$CHAT_TEMPLATE_HOST" ] || die "missing chat template: $CHAT_TEMPLATE_HOST"
+    for r in 1 2 3; do
+        worker_ssh_n "$r" "mkdir -p '$(_tp4_rank_vllm "$r")' '$(_tp4_rank_triton "$r")' '$(_tp4_rank_tilelang "$r")'"
+        _tp4_scp_runtime "$r"
+    done
+    # skip the old single-worker scp block
+    true
+    : <<'TP4_SKIP_OLD_SCP'
+    [ -f "$CHAT_TEMPLATE_HOST" ] || die "missing chat template: $CHAT_TEMPLATE_HOST"
+    scp -q -o BatchMode=yes "$CHAT_TEMPLATE_HOST" "${WORKER_SSH}:/tmp/glm53-chat_template.jinja"
+    [ -f "$VIDEO_PATCH_HOST" ] || die "missing $VIDEO_PATCH_HOST"
+    scp -q -o BatchMode=yes "$VIDEO_PATCH_HOST" "${WORKER_SSH}:/tmp/patch_glm_video_placeholders.py"
+    [ -f "$STOP_PATCH_HOST" ] || die "missing $STOP_PATCH_HOST"
+    scp -q -o BatchMode=yes "$STOP_PATCH_HOST" "${WORKER_SSH}:/tmp/patch_suppress_stops_in_reasoning.py"
+    [ -f "$SCHED_PATCH_HOST" ] || die "missing $SCHED_PATCH_HOST"
+    scp -q -o BatchMode=yes "$SCHED_PATCH_HOST" "${WORKER_SSH}:/tmp/patch_scheduler_decode_floor.py"
+    [ -f "$DRAFTER_PATCH_HOST" ] || die "missing $DRAFTER_PATCH_HOST"
+    scp -q -o BatchMode=yes "$DRAFTER_PATCH_HOST" "${WORKER_SSH}:/tmp/patch_glm5_drafter_group.py"
+    [ -f "$APC_PATCH_HOST" ] || die "missing $APC_PATCH_HOST"
+    scp -q -o BatchMode=yes "$APC_PATCH_HOST" "${WORKER_SSH}:/tmp/patch_hybrid_prefix_hit.py"
+    [ -f "$XGRAMMAR_PATCH_HOST" ] || die "missing $XGRAMMAR_PATCH_HOST"
+    scp -q -o BatchMode=yes "$XGRAMMAR_PATCH_HOST" "${WORKER_SSH}:/tmp/patch_xgrammar_termination.py"
+    [ -f "$KPOOL_TAIL_PATCH_HOST" ] || die "missing $KPOOL_TAIL_PATCH_HOST"
+    scp -q -o BatchMode=yes "$KPOOL_TAIL_PATCH_HOST" "${WORKER_SSH}:/tmp/patch_kpool_tail_slotmap.py"
+    [ -f "$SPINWAIT_PATCH_HOST" ] || die "missing $SPINWAIT_PATCH_HOST"
+    scp -q -o BatchMode=yes "$SPINWAIT_PATCH_HOST" "${WORKER_SSH}:/tmp/patch_spinwait.py"
+
+    worker_ssh "rm -rf /tmp/glm53-ablit"
+    scp -q -r -o BatchMode=yes "$SCRIPT_DIR/ablit" "${WORKER_SSH}:/tmp/glm53-ablit"
+    scp -q -o BatchMode=yes "$SCRIPT_DIR/overlay/ablit_runtime.py" "${WORKER_SSH}:/tmp/glm53-ablit_runtime.py"
+    scp -q -o BatchMode=yes "$SCRIPT_DIR/overlay/patch_ablit.py" "${WORKER_SSH}:/tmp/patch_ablit.py"
+TP4_SKIP_OLD_SCP
+
+    local -a nccl_common=(
+        -e NCCL_IB_DISABLE=0
+        -e NCCL_IB_ROCE_VERSION_NUM=2
+        -e NCCL_NET=IB
+        -e NCCL_NET_PLUGIN=none
+        -e NCCL_NVLS_ENABLE=0
+        -e NCCL_CUMEM_ENABLE=0
+        -e NCCL_IB_MERGE_NICS=0
+        -e "NCCL_CROSS_NIC=$NCCL_CROSS_NIC"
+        -e NCCL_IGNORE_CPU_AFFINITY=1
+        -e "NCCL_DEBUG=$NCCL_DEBUG"
+        -e HF_HUB_OFFLINE=1
+        -e TRANSFORMERS_OFFLINE=1
+        -e HF_HOME=/root/.cache/huggingface
+        -e VLLM_CACHE_ROOT=/root/.cache/vllm
+        -e "GLM53_SUPPRESS_STOPS_IN_REASONING=$GLM53_SUPPRESS_STOPS_IN_REASONING"
+        -e "GLM53_MIXED_PREFILL_CHUNK=$GLM53_MIXED_PREFILL_CHUNK"
+        -e "GLM53_INDEXER_WORKSPACE=$GLM53_INDEXER_WORKSPACE"
+        -e "GLM53_SPINWAIT_MS=$GLM53_SPINWAIT_MS"
+        -e "TRITON_CACHE_DIR=$TRITON_CACHE_DIR"
+        -e "TILELANG_CACHE_DIR=$TILELANG_CACHE_DIR"
+        -e "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=$VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS"
+        -e "TORCH_CUDA_ARCH_LIST=$TORCH_CUDA_ARCH_LIST"
+        -e "FLASHINFER_CUDA_ARCH_LIST=$FLASHINFER_CUDA_ARCH_LIST"
+        -e FLASHINFER_DISABLE_VERSION_CHECK=1
+        -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
+        -e "VLLM_ENGINE_READY_TIMEOUT_S=$READY_TIMEOUT"
+        # py-cpuinfo JSON-parses empty output on Grace/aarch64; the usage
+        # thread then dumps JSONDecodeError. Stats are off on this private kit.
+        -e VLLM_NO_USAGE_STATS=1
+        -e DO_NOT_TRACK=1
+        -e "VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=$CG_ESTIMATE"
+    )
+    local worker_nccl="" e
+    for e in "${nccl_common[@]}"; do
+        [ "$e" = "-e" ] && continue
+        worker_nccl+=" -e $e"
+    done
+
+    local -a head_preload=()
+    if [ "$USE_HOST_NCCL" = "1" ]; then
+        if [ -f "$NCCL_HOST_DIR/$NCCL_SO_NAME" ]; then
+            head_preload=(-v "$NCCL_HOST_DIR:/nccl:ro" -e "LD_PRELOAD=/nccl/$NCCL_SO_NAME")
+            log "head: LD_PRELOAD $NCCL_SO_NAME"
+        else
+            warn "head: $NCCL_HOST_DIR/$NCCL_SO_NAME missing — using image NCCL"
+        fi
+    fi
+
+    local serve_env=""
+    local v
+    for v in SERVED_MODEL_NAME PORT TP NNODES HEAD_IP MASTER_PORT QUANTIZATION \
+             MAX_MODEL_LEN GPU_MEM_UTIL MAX_NUM_SEQS MAX_NUM_BATCHED_TOKENS \
+             KV_CACHE_DTYPE MTP_TOKENS SPEC_METHOD DFLASH_TOKENS DFLASH_MODEL_DIR \
+             DFLASH_DRAFT_TP \
+             LANGUAGE_MODEL_ONLY SKIP_MM_PROFILING \
+             LIMIT_MM CHAT_TEMPLATE ENFORCE_EAGER EXL3_FUSED_MOE EXL3_MOE_ROW_TILE EXL3_TEMP_ROWS_FUSED EXL3_FAT_SORTED EXL3_FAT_BATCHED EXL3_FAT_KERNEL MODEL_DIR EXTRA_ARGS \
+             ABLIT ABLIT_METHOD ABLIT_DIRECTION ABLIT_LAYERS ABLIT_ALPHA ABLIT_INCLUDE_MTP; do
+        serve_env+=" -e $v='${!v:-}'"
+    done
+    # VLLM_API_KEY is read by the head (rank 0) API server for bearer auth; the
+    # worker runs --headless so it only needs the var for argv-parity, and
+    # start.sh below passes it explicitly on the head. Keep it out of the
+    # generic loop so the key never shows in process listings of either node
+    # beyond the container env (same as the DeepSeek deployment).
+    serve_env+=" -e VLLM_API_KEY='${VLLM_API_KEY:-}'"
+
+    local worker_preload="" nccl_dir cname
+    for r in 1 2 3; do
+        worker_preload=""
+        if [ "$USE_HOST_NCCL" = "1" ]; then
+            nccl_dir="$(_tp4_rank_nccl_dir "$r")"
+            if worker_ssh_n "$r" "test -f '$nccl_dir/$NCCL_SO_NAME'"; then
+                worker_preload="-v '$nccl_dir:/nccl:ro' -e LD_PRELOAD='/nccl/$NCCL_SO_NAME'"
+                log "rank ${r}: LD_PRELOAD $NCCL_SO_NAME"
+            else
+                warn "rank ${r}: $nccl_dir/$NCCL_SO_NAME missing — using image NCCL"
+            fi
+        fi
+        cname="$(_tp4_rank_container "$r")"
+        log "starting rank ${r} on $(_tp4_ssh_target "$r") (NCCL if=$(_tp4_rank_cx7_if "$r") hca=$(_tp4_rank_cx7_ib "$r")) ..."
+        [ -n "${LOCAL_MODEL_DIR:-}" ] && worker_preload="$worker_preload -v '$LOCAL_MODEL_DIR:/model:ro'"
+        [ -n "${LOCAL_DFLASH_DIR:-}" ] && worker_preload="$worker_preload -v '$LOCAL_DFLASH_DIR:/draft:ro'"
+        worker_ssh_n "$r" "docker run -d --name '$cname' \
+            --gpus all --network host --ipc=host --shm-size 32g --stop-timeout 60 \
+            --device /dev/infiniband --cap-add IPC_LOCK \
+            --ulimit memlock=-1 --ulimit stack=67108864 --ulimit nofile=1048576:1048576 \
+            -v '$(_tp4_rank_hf "$r"):/root/.cache/huggingface' \
+            -v '$(_tp4_rank_vllm "$r"):/root/.cache/vllm' \
+            -v '$(_tp4_rank_triton "$r"):/root/.triton/cache' \
+            -v '$(_tp4_rank_tilelang "$r"):/root/.tilelang/cache' \
+            -v '/tmp/${cname}.sh:/start.sh:ro' \
+            -v '/tmp/glm53-chat_template.jinja:${CHAT_TEMPLATE}:ro' \
+            -v '/tmp/patch_glm_video_placeholders.py:/opt/glm53/patch_glm_video_placeholders.py:ro' \
+            -v '/tmp/patch_suppress_stops_in_reasoning.py:/opt/glm53/patch_suppress_stops_in_reasoning.py:ro' \
+            -v '/tmp/patch_scheduler_decode_floor.py:/opt/glm53/patch_scheduler_decode_floor.py:ro' \
+            -v '/tmp/patch_glm5_drafter_group.py:/opt/glm53/patch_glm5_drafter_group.py:ro' \
+            -v '/tmp/patch_hybrid_prefix_hit.py:/opt/glm53/patch_hybrid_prefix_hit.py:ro' \
+            -v '/tmp/patch_xgrammar_termination.py:/opt/glm53/patch_xgrammar_termination.py:ro' \
+            -v '/tmp/patch_kpool_tail_slotmap.py:/opt/glm53/patch_kpool_tail_slotmap.py:ro' \
+            -v '/tmp/patch_spinwait.py:/opt/glm53/patch_spinwait.py:ro' \
+            -v '${ABLIT_HOST_DIR:-/tmp/glm53-ablit}:/opt/glm53/ablit:ro' \
+            -v '/tmp/glm53-ablit_runtime.py:/opt/glm53/ablit_runtime.py:ro' \
+            -v '/tmp/patch_ablit.py:/opt/glm53/patch_ablit.py:ro' \
+            ${worker_preload} \
+            ${worker_nccl} \
+            -e NCCL_SOCKET_IFNAME='$(_tp4_rank_cx7_if "$r")' \
+            -e GLOO_SOCKET_IFNAME='$(_tp4_rank_cx7_if "$r")' \
+            -e NCCL_IB_HCA='$(_tp4_rank_cx7_ib "$r")' \
+            -e NCCL_IB_GID_INDEX='$(_tp4_rank_gid "$r")' \
+            -e VLLM_HOST_IP='$(_tp4_rank_ip "$r")' \
+            -e NODE_RANK='$r' \
+            ${serve_env} \
+            --entrypoint bash '$IMAGE' /start.sh" >/dev/null
+    done
+
+    log "starting head (vLLM API :${PORT}; NCCL if=${HEAD_CX7_IF} hca=${HEAD_CX7_IB}) ..."
+    [ -n "${LOCAL_MODEL_DIR:-}" ] && head_preload+=(-v "$LOCAL_MODEL_DIR:/model:ro")
+    [ -n "${LOCAL_DFLASH_DIR:-}" ] && head_preload+=(-v "$LOCAL_DFLASH_DIR:/draft:ro")
+    docker run -d --name "$CONTAINER_HEAD" \
+        --gpus all --network host --ipc=host --shm-size 32g --stop-timeout 60 \
+        --device /dev/infiniband --cap-add IPC_LOCK \
+        --ulimit memlock=-1 --ulimit stack=67108864 --ulimit nofile=1048576:1048576 \
+        -v "$HF_CACHE_DIR:/root/.cache/huggingface" \
+        -v "$CACHE_ROOT:/root/.cache/vllm" \
+        -v "$TRITON_HOST_CACHE:/root/.triton/cache" \
+        -v "$TILELANG_HOST_CACHE:/root/.tilelang/cache" \
+        -v "$HEAD_SCRIPT:/start.sh:ro" \
+        -v "$CHAT_TEMPLATE_HOST:$CHAT_TEMPLATE:ro" \
+        -v "$VIDEO_PATCH_HOST:/opt/glm53/patch_glm_video_placeholders.py:ro" \
+        -v "$STOP_PATCH_HOST:/opt/glm53/patch_suppress_stops_in_reasoning.py:ro" \
+        -v "$SCHED_PATCH_HOST:/opt/glm53/patch_scheduler_decode_floor.py:ro" \
+        -v "$DRAFTER_PATCH_HOST:/opt/glm53/patch_glm5_drafter_group.py:ro" \
+        -v "$APC_PATCH_HOST:/opt/glm53/patch_hybrid_prefix_hit.py:ro" \
+        -v "$XGRAMMAR_PATCH_HOST:/opt/glm53/patch_xgrammar_termination.py:ro" \
+        -v "$KPOOL_TAIL_PATCH_HOST:/opt/glm53/patch_kpool_tail_slotmap.py:ro" \
+        -v "$SPINWAIT_PATCH_HOST:/opt/glm53/patch_spinwait.py:ro" \
+        -v "${ABLIT_HOST_DIR:-$SCRIPT_DIR/ablit}:/opt/glm53/ablit:ro" \
+        -v "$SCRIPT_DIR/overlay/ablit_runtime.py:/opt/glm53/ablit_runtime.py:ro" \
+        -v "$SCRIPT_DIR/overlay/patch_ablit.py:/opt/glm53/patch_ablit.py:ro" \
+        "${head_preload[@]}" \
+        "${nccl_common[@]}" \
+        -e NCCL_SOCKET_IFNAME="$HEAD_CX7_IF" \
+        -e GLOO_SOCKET_IFNAME="$HEAD_CX7_IF" \
+        -e NCCL_IB_HCA="$HEAD_CX7_IB" \
+        -e NCCL_IB_GID_INDEX="$HEAD_GID" \
+        -e VLLM_HOST_IP="$HEAD_IP" \
+        -e SERVED_MODEL_NAME="$SERVED_MODEL_NAME" \
+        -e PORT="$PORT" -e TP="$TP" -e NNODES="$NNODES" \
+        -e HEAD_IP="$HEAD_IP" -e MASTER_PORT="$MASTER_PORT" \
+        -e QUANTIZATION="$QUANTIZATION" \
+        -e MAX_MODEL_LEN="$MAX_MODEL_LEN" -e GPU_MEM_UTIL="$GPU_MEM_UTIL" \
+        -e MAX_NUM_SEQS="$MAX_NUM_SEQS" \
+        -e MAX_NUM_BATCHED_TOKENS="$MAX_NUM_BATCHED_TOKENS" \
+        -e KV_CACHE_DTYPE="$KV_CACHE_DTYPE" -e MTP_TOKENS="$MTP_TOKENS" \
+        -e SPEC_METHOD="$SPEC_METHOD" \
+        -e DFLASH_TOKENS="${DFLASH_TOKENS:-7}" \
+        -e DFLASH_MODEL_DIR="${DFLASH_MODEL_DIR:-}" \
+        -e DFLASH_DRAFT_TP="${DFLASH_DRAFT_TP:-}" \
+        -e LANGUAGE_MODEL_ONLY="$LANGUAGE_MODEL_ONLY" \
+        -e SKIP_MM_PROFILING="$SKIP_MM_PROFILING" \
+        -e LIMIT_MM="$LIMIT_MM" \
+        -e CHAT_TEMPLATE="$CHAT_TEMPLATE" \
+        -e ENFORCE_EAGER="$ENFORCE_EAGER" \
+        -e EXL3_FUSED_MOE="$EXL3_FUSED_MOE" \
+        -e EXL3_MOE_ROW_TILE="$EXL3_MOE_ROW_TILE" \
+        -e EXL3_TEMP_ROWS_FUSED="$EXL3_TEMP_ROWS_FUSED" \
+        -e EXL3_FAT_SORTED="$EXL3_FAT_SORTED" \
+        -e EXL3_FAT_BATCHED="$EXL3_FAT_BATCHED" \
+        -e EXL3_FAT_KERNEL="$EXL3_FAT_KERNEL" \
+        -e ABLIT="$ABLIT" \
+        -e ABLIT_METHOD="$ABLIT_METHOD" \
+        -e ABLIT_DIRECTION="$ABLIT_DIRECTION" \
+        -e ABLIT_LAYERS="$ABLIT_LAYERS" \
+        -e ABLIT_ALPHA="$ABLIT_ALPHA" \
+        -e ABLIT_INCLUDE_MTP="$ABLIT_INCLUDE_MTP" \
+        -e MODEL_DIR="$MODEL_DIR" \
+        -e VLLM_API_KEY="$VLLM_API_KEY" \
+        -e EXTRA_ARGS="${EXTRA_ARGS:-}" \
+        --entrypoint bash "$IMAGE" /start.sh >/dev/null
+
+    log "containers up — head=${CONTAINER_HEAD}, workers=${CONTAINER_WORKER} ${CONTAINER_WORKER2} ${CONTAINER_WORKER3}"
+}
+
+# ---------------------------- health wait ----------------------------------
+wait_for_health() {
+    local url="http://127.0.0.1:${PORT}/health"
+    log "waiting for ${url} (weight load + warmup on a 320B MoE is slow; timeout ${READY_TIMEOUT}s) ..."
+    log "streaming head logs live — Ctrl-C detaches, the server keeps running"
+
+    local logpid=""
+    _stop_logtail() {
+        [ -n "$logpid" ] && kill "$logpid" 2>/dev/null || true
+        wait "$logpid" 2>/dev/null || true
+        logpid=""
+    }
+    trap '_stop_logtail; warn "interrupted — containers keep running ('"'"'./start-tp4.sh logs'"'"' / '"'"'./start-tp4.sh stop'"'"')"; exit 130' INT
+    docker logs -f --tail 0 "$CONTAINER_HEAD" 2>&1 &
+    logpid=$!
+
+    local elapsed=0 healthy=0 exited=0 dead_side="" worker_fail=0
+    while [ "$elapsed" -lt "$READY_TIMEOUT" ]; do
+        if curl -fsS -m 5 "$url" >/dev/null 2>&1; then healthy=1; break; fi
+        if ! docker inspect -f '{{.State.Running}}' "$CONTAINER_HEAD" 2>/dev/null | grep -q true; then
+            log "head container exited during startup"
+            exited=1; dead_side="head"; break
+        fi
+        # A dead worker rank can never make the head healthy — fail fast with
+        # the log dump instead of polling for the full READY_TIMEOUT (issue
+        # #22, item 4). Transient ssh/docker hiccups are tolerated; only
+        # three consecutive non-running answers (~30 s) count as a dead
+        # worker.
+        local all_up=1 wr
+        for wr in 1 2 3; do
+            if worker_ssh_n "$wr" "docker inspect -f '{{.State.Running}}' '$(_tp4_rank_container "$wr")' 2>/dev/null" | grep -q true; then
+                :
+            else
+                all_up=0
+                dead_side="rank${wr}"
+            fi
+        done
+        if [ "$all_up" = "1" ]; then
+            worker_fail=0
+        else
+            worker_fail=$((worker_fail + 1))
+            if [ "$worker_fail" -ge 3 ]; then
+                log "a worker container is not running (${dead_side}, 3 consecutive checks)"
+                exited=1; break
+            fi
+        fi
+        sleep 10; elapsed=$((elapsed + 10))
+    done
+
+    _stop_logtail
+    trap 'warn "interrupted — containers keep running ('"'"'./start-tp4.sh logs'"'"' / '"'"'./start-tp4.sh stop'"'"')"; exit 130' INT
+
+    if [ "$healthy" = "1" ]; then
+        log "health check passed after ${elapsed}s — server is up"
+    elif [ "$exited" = "1" ]; then
+        warn "${dead_side:-head} container exited/stopped after ${elapsed}s"
+    else
+        warn "timed out after ${elapsed}s without becoming healthy"
+    fi
+    [ "$healthy" = "1" ]
+}
+
+post_ready_warmup() {
+    if [ "${GLM53_BOOT_SHAPE_WARMUP:-1}" = "0" ]; then
+        log "boot shape warmup skipped (GLM53_BOOT_SHAPE_WARMUP=0)"
+        return 0
+    fi
+    [ -f "$SCRIPT_DIR/scripts/boot-shape-warmup.sh" ] \
+        || { warn "boot-shape-warmup.sh missing — skipping"; return 0; }
+    log "post-ready DFlash2/sampler warmup (nonfatal; timeout ${GLM53_WARMUP_REQ_TIMEOUT}s/req) ..."
+    GLM53_WARMUP_MAX_CONCURRENCY="$MAX_NUM_SEQS" \
+    GLM53_WARMUP_REQ_TIMEOUT="$GLM53_WARMUP_REQ_TIMEOUT" \
+    GLM53_WARMUP_DFLASH_K="${DFLASH_TOKENS:-7}" \
+    GLM53_WARMUP_TRITON_CACHE_DIR="$TRITON_HOST_CACHE" \
+    GLM53_WARMUP_BEARER="${VLLM_API_KEY:-}" \
+        bash "$SCRIPT_DIR/scripts/boot-shape-warmup.sh" \
+            "http://127.0.0.1:${PORT}" "$SERVED_MODEL_NAME" \
+        || warn "boot shape warmup incomplete — uncovered shapes may JIT mid-serve on TP=4"
+}
+
+collect_failure_logs() {
+    mkdir -p "$LOGDIR"
+    docker logs "$CONTAINER_HEAD" >"$LOGDIR/head.log" 2>&1 || true
+    local r
+    for r in 1 2 3; do
+        worker_ssh_n "$r" "docker logs '$(_tp4_rank_container "$r")' 2>&1" >"$LOGDIR/worker${r}.log" 2>&1 || true
+    done
+}
+
+on_ready() {
+    log "======================================================================"
+    log "GLM-5.3-Flash EXL3 is UP (TP=${TP}, nnodes=${NNODES})"
+    log "  endpoints  : http://127.0.0.1:${PORT}/v1   (LAN: ${HEAD_IP}:${PORT})"
+    log "  model name : ${SERVED_MODEL_NAME}"
+    log "  weights    : ${MODEL}  quant=${QUANTIZATION}  kv=${KV_CACHE_DTYPE}"
+    local vision=on
+    [ "${LANGUAGE_MODEL_ONLY}" = "1" ] && vision=off
+    local spec="MTP k=${MTP_TOKENS}"
+    [ "$SPEC_METHOD" = "dflash" ] && spec="DFlash2 k=${DFLASH_TOKENS} (${DFLASH_MODEL})"
+    [ "$SPEC_METHOD" = "none" ] && spec=off
+    local ablit="off (stock weights)"
+    [ "$ABLIT" = "1" ] && ablit="ON method=${ABLIT_METHOD} direction=${ABLIT_DIRECTION} layers=${ABLIT_LAYERS} alpha=${ABLIT_ALPHA}"
+    log "  features   : tools=glm47+auto, reasoning=glm45, spec=${spec}, vision=${vision}, ablit=${ablit}"
+    local auth_line="none (VLLM_API_KEY empty)"
+    if [ -n "${VLLM_API_KEY:-}" ]; then
+        auth_line="bearer token set (VLLM_API_KEY) — send Authorization: Bearer <key> on /v1 requests"
+    fi
+    log "  auth       : ${auth_line}"
+    log "  quick test :"
+    log "    curl -s http://127.0.0.1:${PORT}/v1/chat/completions \\"
+    if [ -n "${VLLM_API_KEY:-}" ]; then
+        log "      -H 'Authorization: Bearer <KEY>' \\"
+    fi
+    log "      -H 'Content-Type: application/json' \\"
+    log "      -d '{\"model\": \"${SERVED_MODEL_NAME}\", \"messages\": [{\"role\": \"user\", \"content\": \"hello!\"}]}'"
+    log "  manage     : ./start-tp4.sh status | ./start-tp4.sh logs | ./start-tp4.sh logs 1 | ./start-tp4.sh stop"
+    log "======================================================================"
+    if [ "${TAIL:-0}" = "1" ]; then
+        log "tailing head logs — Ctrl-C just detaches, the server keeps running"
+        trap '' INT
+        docker logs -f --tail 20 "$CONTAINER_HEAD" || true
+        trap 'warn "interrupted — containers keep running"; exit 130' INT
+    fi
+}
+
+# ------------------------------- start -------------------------------------
+start() {
+    preflight
+    ensure_image
+    download_weights
+    download_dflash
+    sync_weights
+    write_inner_scripts
+
+    if [ -n "${LOCAL_MODEL_DIR:-}" ]; then
+        MODEL_DIR="/model"
+        log "LOCAL_MODEL_DIR mode: $LOCAL_MODEL_DIR -> /model on all ranks"
+    else
+        MODEL_DIR="$(resolve_model_dir)"
+    fi
+    DFLASH_MODEL_DIR=""
+    if [ "$SPEC_METHOD" = "dflash" ]; then
+        if [ -n "${LOCAL_DFLASH_DIR:-}" ]; then
+            DFLASH_MODEL_DIR="/draft"
+        else
+            DFLASH_MODEL_DIR="$(resolve_dflash_dir)"
+        fi
+        log "DFlash2 load path (in-container): ${DFLASH_MODEL_DIR}"
+    fi
+    log "model load path (in-container): ${MODEL_DIR}"
+    log "config: image=${IMAGE} tp=${TP} nnodes=${NNODES} quant=${QUANTIZATION} spec=${SPEC_METHOD} mtp=${MTP_TOKENS} dflash_k=${DFLASH_TOKENS} max-len=${MAX_MODEL_LEN} gpu-util=${GPU_MEM_UTIL} kv=${KV_CACHE_DTYPE} lm-only=${LANGUAGE_MODEL_ONLY} port=${PORT}"
+
+    launch_cluster
+    if wait_for_health; then
+        post_ready_warmup
+        on_ready
+        return
+    fi
+    collect_failure_logs
+    echo "---- last 60 lines of head log ($LOGDIR/head.log) ----"
+    tail -n 60 "$LOGDIR/head.log" || true
+    echo "---- last 40 lines of worker1 log ($LOGDIR/worker1.log) ----"
+    tail -n 40 "$LOGDIR/worker1.log" || true
+    echo "---- last 40 lines of worker2 log ($LOGDIR/worker2.log) ----"
+    tail -n 40 "$LOGDIR/worker2.log" || true
+    echo "---- last 40 lines of worker3 log ($LOGDIR/worker3.log) ----"
+    tail -n 40 "$LOGDIR/worker3.log" || true
+    die "server did not become healthy — full logs in $LOGDIR/"
+}
+
+# ------------------------------- stop --------------------------------------
+stop() {
+    local r
+    log "stopping head container ..."
+    docker rm -f "$CONTAINER_HEAD" >/dev/null 2>&1 || log "  (no head container was running)"
+    for r in 1 2 3; do
+        log "stopping rank ${r} on $(_tp4_ssh_target "$r") ..."
+        worker_ssh_n "$r" "docker rm -f '$(_tp4_rank_container "$r")'" >/dev/null 2>&1 \
+            || log "  (no rank ${r} container was running)"
+    done
+    log "stopped."
+}
+
+# ------------------------------ status -------------------------------------
+status() {
+    log "head (${CONTAINER_HEAD} on $(hostname)):"
+    docker ps -a --filter "name=${CONTAINER_HEAD}" --format '  {{.Names}}  {{.Status}}' || true
+    if curl -fsS -m 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then
+        log "  API: healthy — http://127.0.0.1:${PORT}/v1"
+    else
+        log "  API: not responding"
+    fi
+    local r
+    for r in 1 2 3; do
+        log "rank ${r} ($(_tp4_rank_container "$r") on $(_tp4_ssh_target "$r")):"
+        worker_ssh_n "$r" "docker ps -a --filter name=$(_tp4_rank_container "$r") --format '  {{.Names}}  {{.Status}}'" 2>/dev/null \
+            || log "  (rank ${r} unreachable)"
+    done
+}
+
+# ------------------------------- logs --------------------------------------
+logs() {
+    case "${1:-head}" in
+        1|2|3|worker|worker1)
+            local r="${1}"
+            if [ "$r" = "worker" ] || [ "$r" = "worker1" ]; then r=1; fi
+            log "following rank ${r} logs on $(_tp4_ssh_target "$r") ..."
+            trap '' INT
+            worker_ssh_n "$r" "docker logs -f --tail 100 '$(_tp4_rank_container "$r")'" || true
+            trap 'warn "interrupted"; exit 130' INT
+            ;;
+        head|*)
+            log "following head logs (driver + API server) ..."
+            trap '' INT
+            docker logs -f --tail 100 "$CONTAINER_HEAD" || true
+            trap 'warn "interrupted"; exit 130' INT
+            ;;
+    esac
+}
+
+# ------------------------------- main --------------------------------------
+main() {
+    local cmd="${1:-start}"
+    case "$cmd" in
+        start|restart) validate_numeric_config ;;
+    esac
+    case "$cmd" in
+        stop)     banner stop.sh ;;
+        download) banner download.sh ;;
+        *)        banner start-tp4.sh ;;
+    esac
+    case "$cmd" in
+        start)    shift || true; start ;;
+        download) download_only ;;
+        stop)     stop ;;
+        restart)  stop; start ;;
+        status)   status ;;
+        logs)     shift || true; logs "$@" ;;
+        -h|--help|help) usage ;;
+        *) usage; exit 1 ;;
+    esac
+}
+
+main "$@"

serving/taskbench.py

@@ -0,0 +1,52 @@
+#!/usr/bin/env python3
+"""Task-class x concurrency decode benchmark for the AEON serve (OpenAI chat API, vLLM metrics).
+gen tok/s = completion tokens / (wall - TTFT) per stream; aggregate = sum tokens / max wall (decode window).
+tok/pass = delta(spec accepted)/delta(drafts) + 1 from /metrics."""
+import json, os, sys, time, threading, urllib.request, statistics, random
+BASE=os.environ.get("BASE","http://localhost:8000"); MODEL=os.environ.get("MODEL","aeon"); OUT=int(os.environ.get("OUT","512"))
+THINK=os.environ.get("THINK","0")=="1"
+TOPICS=["a lighthouse keeper in 1890s Norway","a courier robot in a flooded Bangkok","a violinist who loses her hearing","a mountain village's last winter","two brothers repairing a boat","a night-shift nurse in Lagos","a chess prodigy in exile","a beekeeper during a drought"]
+TASKS={
+ "prose": lambda i: f"Write a vivid literary short story (at least 700 words) about {TOPICS[i%8]}. No headings, flowing prose only.",
+ "code":  lambda i: f"Write a complete, well-documented Python module implementing an LRU cache with TTL expiry, thread safety, statistics, and a small CLI demo. Variant #{i}: name the class Cache{i}. Code only, no prose.",
+ "list":  lambda i: f"List 120 distinct {['programming concepts','world cities','chemical compounds','historical events','musical instruments','mathematical theorems','plant species','cooking techniques'][i%8]}, numbered, each with a one-line description.",
+ "essay": lambda i: f"Write a structured argumentative essay (about 800 words, with an introduction, three sections with headers, and a conclusion) on: '{['Should cities ban private cars?','Is remote work good for innovation?','Should AI systems have a right to refuse tasks?','Is nuclear power essential for decarbonization?','Should voting be mandatory?','Do social networks make us lonelier?','Should space exploration be privatized?','Is a four-day work week viable?'][i%8]}'",
+}
+def metrics():
+    t=urllib.request.urlopen(BASE+"/metrics",timeout=10).read().decode(); d={}
+    for l in t.splitlines():
+        for k in ("spec_decode_num_drafts_total","spec_decode_num_accepted_tokens_total"):
+            if l.startswith("vllm:"+k+"{"): d[k]=float(l.rsplit(" ",1)[1])
+    return d
+def one(prompt, res, idx):
+    body={"model":MODEL,"messages":[{"role":"user","content":prompt}],"max_tokens":OUT,"stream":True,
+          "stream_options":{"include_usage":True},"chat_template_kwargs":{"enable_thinking":THINK}}
+    body.update(json.loads(os.environ.get("SAMPLING","{}")))
+    req=urllib.request.Request(BASE+"/v1/chat/completions",data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
+    t0=time.time(); ttft=None; ntok=0
+    with urllib.request.urlopen(req,timeout=1800) as r:
+        for line in r:
+            line=line.decode().strip()
+            if not line.startswith("data:") or line=="data: [DONE]": continue
+            ev=json.loads(line[5:])
+            if ev.get("usage"): ntok=ev["usage"]["completion_tokens"]
+            ch=ev.get("choices")
+            d=(ch[0].get("delta") or {}) if ch else {}
+            if ttft is None and (d.get("content") or d.get("reasoning_content") or d.get("reasoning")): ttft=time.time()-t0
+    wall=time.time()-t0; res[idx]=(ntok,ttft or wall,wall)
+def run(task,conc):
+    m0=metrics(); res=[None]*conc; th=[threading.Thread(target=one,args=(TASKS[task](i+conc*7),res,i)) for i in range(conc)]
+    T=time.time(); [t.start() for t in th]; [t.join() for t in th]; W=time.time()-T; m1=metrics()
+    toks=sum(r[0] for r in res); per=[r[0]/(r[2]-r[1]) for r in res if r[2]>r[1]] or [float("nan")]
+    g=lambda m,k: m.get(k,0.0)
+    dr=g(m1,"spec_decode_num_drafts_total")-g(m0,"spec_decode_num_drafts_total"); ac=g(m1,"spec_decode_num_accepted_tokens_total")-g(m0,"spec_decode_num_accepted_tokens_total")
+    tpp=(ac/dr+1) if dr else float("nan"); ttft=statistics.median(r[1] for r in res)
+    return dict(task=task,conc=conc,agg=toks/W,per=statistics.mean(per),ttft=ttft,tpp=tpp,tok=toks/conc)
+if __name__=="__main__":
+    tasks=sys.argv[1].split(",") if len(sys.argv)>1 else list(TASKS); concs=[int(c) for c in (sys.argv[2] if len(sys.argv)>2 else "1,2,4,8").split(",")]
+    run("prose",1)  # warm
+    print(f"thinking={'on' if THINK else 'off'} out={OUT} sampling=server default (temp 0.6/top_p .95/top_k 20/rep 1.05)")
+    print(f"{'task':6} {'C':>2} {'agg t/s':>8} {'per-stream':>10} {'TTFT':>6} {'tok/pass':>8} {'tok/req':>7}")
+    for task in tasks:
+        for c in concs:
+            r=run(task,c); print(f"{r['task']:6} {r['conc']:>2} {r['agg']:8.1f} {r['per']:10.1f} {r['ttft']:6.2f} {r['tpp']:8.2f} {r['tok']:7.0f}",flush=True)

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "[MASK]",
+    "[gMASK]",
+    "[sMASK]",
+    "<sop>",
+    "<eop>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|observation|>",
+    "<|begin_of_image|>",
+    "<|end_of_image|>",
+    "<|begin_of_video|>",
+    "<|end_of_video|>",
+    "<|begin_of_audio|>",
+    "<|end_of_audio|>",
+    "<|begin_of_transcription|>",
+    "<|end_of_transcription|>"
+  ],
+  "is_local": true,
+  "model_max_length": 1048576,
+  "model_specific_special_tokens": {},
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "remove_space": false,
+  "tokenizer_class": "TokenizersBackend"
+}