OpensourceWTF/MiMo-V2.6-Flash-MOPD-fp8res-mlx-stream

MiMo-V2.6-Flash-MOPD, fp8res mlx-stream pack

A serving pack of XiaomiMiMo/MiMo-V2.6-Flash-MOPD for mlx-serve with the mlx-stream plugin on Apple silicon. The routed experts (the bulk of the model) stay on disk and stream from the SSD per request; the attention, dense and router weights stay resident. On one M5 Max (128 GB) this serves the 309B-parameter model at about 1,000 tok/s prefill and 30 tok/s decode for a 16K-token prompt, and holds the full 1,048,576-token context.

The weights are the release's: the routed experts are a bit-exact repack of the release's MXFP4 experts into fixed 13,369,344-byte records (experts.bin), and the resident projections keep the release's FP8 (e4m3, 64x128 blocks). Nothing was re-quantized. License: MIT, inherited from the release; the model is Xiaomi's.

Files (171 GB)

file	bytes	what
experts.bin	160,859,947,008	the expert bank: 47 MoE layers x 256 routed experts, one 13,369,344-byte record each (gate, up, down in the release's MXFP4 group-32 layout), format mlx-stream-mxfp4-expert-bank-v1
expert-manifest.json	2,087,572	per-record offset, length and sha256; the bank's format and geometry
model.safetensors	8,895,389,040	the residents: embeddings, attention, the dense layer, routers, norms, lm_head; q/k/v and the dense MLP in the release's FP8
model.safetensors.index.json	40,989	the residents' index
mtp-heads.safetensors	1,189,599,424	the release's multi-token-prediction heads, the draft lane's sidecar
mtp-manifest.json	23,723	the sidecar's binding to this pack (data sha256, payload sizes)
config.json	84,703	the release config plus the expert_streaming block the plugin reads (residents_format: fp8_block128)
tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json		the release's

experts.bin must be a real file or a hard link on the serving machine (the stream reads it with uncached I/O).

Serving

Requirements: Apple silicon with 128 GB unified memory (the fp8res pack needs about 104-108 GB at a 16K prompt), the pack on the internal SSD, mlx-serve built with the mlx-stream plugin on its main (the host from branch pr/mimo-host until ddalcu/mlx-serve#822 merges), and the wired limit raised:

sudo sysctl iogpu.wired_limit_mb=114688

The command the measurements used (the harness's host/port/timeout flags dropped):

MLX_SERVE_GPU_CEILING_MB=114688 mlx-serve serve --model /abs/path/to/MiMo-V2.6-Flash-MOPD-fp8res-mlx-stream \
  --ctx-size 32768 --no-vision --no-warmup-eager --prefix-cache-entries 0 --metrics --log-file off \
  --kv-quant off --wired-margin-gib 2

--ctx-size is the context the memory bill covers (prompt plus generation); the plugin fills the resident expert rows up to the GPU ceiling less the wired margin and logs the fill (MiMo memory: ... prompt_rows= decode_rows=). Every route is a construction setting in the pack's config.json under expert_streaming.settings; the defaults are the measured best routes, and the measurements below pinned the rows explicitly, for example:

"expert_streaming": { "settings": { "expert_prefill_rows": 130, "expert_rows": 147, "mtp_lane": true } }

The server speaks the OpenAI-compatible API (/v1/completions, /v1/chat/completions). The plugin's docs/mimo.md has the full recipe, every setting, the memory bill and the limitations; docs/mimo-pack.md describes the pack and how it was built from the release.

Measurements

One M5 Max, 128 GB, one internal SSD (about 13.5 GB/s), 2026-10-11, nothing else loaded, each cell a fresh server and one request. KV bf16, the draft (MTP) lane on, FP8 residents. "Card" = temperature 1.0, top_p 0.95, seed 1, thinking on; the standard 16K prompt is a 16,382-token coding prompt; rows = resident expert rows per layer (prompt / decode).

cell	rows	prefill tok/s	decode tok/s	TTFT s	answer
16K card, 16,384-token answer (mean of 2, identical outputs)	134 / 152	999.8	28.25	16.4	16,384 tokens, cap
16K card, 688-token answer	130 / 147	1,003.9	31.6	16.3	complete, stopped on EOS
16K card on the binary of record (same cell, identical output)	130 / 147	998.8	32.3	16.4	complete, stopped on EOS

The card ladder, one cold card request per prompt length, rows pinned per rung to what the memory bill admits:

prompt tokens	rows	prefill tok/s	decode tok/s	TTFT	answer
1,023	130 / 147	92.3 (cold first request)	31.6	11.1 s	EOS
2,048	129 / 146	178.3	33.1	11.5 s	EOS
4,096	129 / 146	345.5	32.2	11.9 s	EOS
8,192	129 / 146	654.4	30.9	12.5 s	EOS
16,384	129 / 146	978.5	30.7	16.7 s	EOS
32,768	128 / 144	827.8	31.3	39.6 s	EOS
65,536	125 / 142	743.9	29.1	88.1 s	EOS
131,072	121 / 139	599.9	25.7	3.6 min	EOS
262,144	114 / 133	439.9	21.3	9.9 min	EOS
524,287	98 / 120	296.1	15.4	29.5 min	EOS
1,032,192	67 / 96	179.6	9.6	95.8 min	EOS

Every rung returned a complete, correct answer to the same five-function coding task buried behind the context.

The other per-metric charts: time to first token, MLX allocator peak, process footprint peak, completion tokens. The tables with rows per rung and the provenance of every cell are in the plugin's docs/benchmarks/mimo-v26-flash/README.md.

HumanEval (164 problems, one sample each, pass@1, rows 127 / 145; greedy = temperature 0, thinking off, 8,192-token cap; card = temperature 1.0, top_p 0.95, seed 1, thinking on, 16,384-token cap): greedy 155 / 164 = 0.9451, card 157 / 164 = 0.9573; every failure is a wrong answer, none truncated, looped, without code or errored.

Why a dynamic expert cache: within one context the busiest half of a layer's experts takes about 85% of the routes, but which half changes with the context, so the plugin's dynamic residency (59% of the bank resident) hits 94.5% of decode routes where a static hot set of the prompt's top experts would hit 50-58%.

Provenance

Built from XiaomiMiMo/MiMo-V2.6-Flash-MOPD at revision 2479e2d0029eca9a34cc7e7f55a121925f81908e: the 128 expert shards (171,743,035,008 bytes, each sha256-checked against its LFS object) repacked record by record into experts.bin (parity-checked bit-exact on sampled records; every record's sha256 is in expert-manifest.json), the residents converted to one safetensors file with the release's FP8 projections kept, the MTP heads cut into the sidecar. config.json, the tokenizer files and the chat template are the release's with the expert_streaming block added. The build steps are listed in the plugin's docs/mimo-pack.md.

Measurements and software: mlx-serve host pr/mimo-host (2c041af6) with the mlx-stream plugin at 05fc760 (binary sha256 4b018c54...); the ladder and HumanEval ran on the predecessor build (plugin 8e7d186), which the binary of record reproduces token for token on the 16K cell. Charts: scripts/mimo_charts.py in the plugin.