Maximilian228/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Strata

NVIDIA's Qwen3.8-Flash-Next-NVFP4, converted for Strata

Quantized by NVIDIA. These are the weights of NVIDIA's checkpoint nvidia/Qwen3.8-Flash-Next-NVFP4 (NVIDIA Model Optimizer), unchanged. This repository only converts their file format for Strata NVFP4. It is not made or endorsed by NVIDIA.

NVIDIA's NVFP4 checkpoint of Qwen's original Qwen3.8-Flash-Next, converted for Strata NVFP4: the 125B hybrid MoE on one RTX 20-50 card (12 GB of VRAM or more) and 64 GB of RAM or more. NVIDIA's checkpoint is made for vLLM on B200/B300 servers (8 GPUs in its examples); these are the same weights in the files Strata's engine reads, so it runs at full speed on a desktop.

How it compares

Model	Same top token as Q8_0	Decode, tokens/s	32K prompt	RAM in use	PC for this speed	Get it
Q8_0: every expert Q8_0 (ours, qwen-q8)	100% (the reference)	47	10.3 s	106 GiB	128 GB	Maximilian228/Qwen3.8-Flash-Next-Q8_0-Strata
NVFP4 + Q8_0 mix (ours, qwen-nvfp4-mix)	95.0%	91	4.2 s	105 GiB	128 GB	Maximilian228/Qwen3.8-Flash-Next-NVFP4-Q8-Mix-Strata
Unsloth UD-Q4_K_XL (unsloth, --model UD-Q4_K_XL)	94.1%	169	4.9 s	77 GiB	96 GB	unsloth/Qwen3.8-Flash-Next-GGUF
NVFP4 GPTQ (ours, qwen-nvfp4-gptq, setup's default)	93.6%	177	3.9 s	69 GiB	96 GB	Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata
This model: NVIDIA NVFP4, converted by us (qwen-nvidia-nvfp4)	92.9%	167	3.9 s	69 GiB	96 GB	Maximilian228/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Strata
NVIDIA FP4-Local, retrained by NVIDIA for FP4 (QAD), converted on the PC	92.2%	191¹	3.7 s¹	69 GiB	96 GB	nvidia/Qwen3.8-Flash-Next-FP4-Local
Unsloth UD-Q3_K_XL (unsloth, --model UD-Q3_K_XL)	90.7%	208	3.8 s	58 GiB	96 GB	unsloth/Qwen3.8-Flash-Next-GGUF
ISTA-DASLab IQ3_S (qwen, --model IQ3_S)	90.0%	228	3.9 s	52 GiB	64 GB	ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
ISTA-DASLab IQ3_XXS (qwen, --model IQ3_XXS, upstream's pick)	87.3%	239	3.9 s	45 GiB	64 GB	ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
ISTA-DASLab IQ2_XS (qwen, --model IQ2_XS)	85.0%	265	4.2 s	38 GiB	48 GB	ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
ISTA-DASLab Q2_0 (qwen, --model Q2_0)	84.1%	256	3.9 s	37 GiB	48 GB	ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF

• Accuracy: how often the model's most likely next token is the Q8_0 model's. Measured on 5,153 tokens of code, docs, English and Ukrainian chats and agent transcripts, every model fed the same text. Every model ran with its experts on the CPU and the route tail skip off, so only the quantization differs. The Q8_0 model also has Q8_0 attention, Qwen's BF16 router and Qwen's own FP8 PLE, so it is the original's precision everywhere but the 8-bit weights.
• Speed: RTX 5090 (32 GB), Ryzen 9 9950X3D, 128 GB of DDR5-5600, Windows 11, Strata NVFP4 0.1.42-nvfp4.1's engine with the arguments 0.1.42-nvfp4.2's setup writes for each model. Median of 3 runs: 4 greedy chats (up to 1,500 tokens each) and one 32,007-token prompt.
• RAM: "RAM in use" is the engine and the server at the speed shown; "PC for this speed" is the installed RAM that holds it with Windows beside it. Smaller PCs still run the models, slower: the NVFP4 GPTQ and NVIDIA's on a 64 GB PC in the low-RAM mode decode ~115 tokens/s and read a 32K prompt in ~9 s (docs/NVFP4.md, "64 GB of RAM"); Unsloth's files go into a RAM-budget mode there and read experts from the SSD (UD-Q4_K_XL: 7-8.5 tokens/s at 64 GB, upstream's measurement); setup's minimum is 48 GB for them and the ISTA 2-bit files, 60-64 GB for the ISTA 3-bit ones.
• Unsloth's files keep every expert in RAM where they fit with 16 GiB more (setup from 0.1.42-nvfp4.2): from ~88 GB of RAM for UD-Q4_K_XL and ~68 GB for UD-Q3_K_XL. Below that, setup's RAM-budget mode reads the rest from the GGUF on the SSD. Setup before 0.1.42-nvfp4.2 used the budget mode on every PC; on 128 GB UD-Q4_K_XL then decoded 118 tokens/s and read the 32K prompt in 30 s (run setup again to switch an existing install).
• UD-Q3_K_XL's experts are i-quants despite the name: IQ3_XXS gate/up with IQ4_NL or Q8_0 down.
• The GGUF models run upstream's route tail skip by default: ~10% faster decode, and answers a little further from Q8_0 than the table shows.
• The two 128 GB models need a 128 GB PC (setup asks below that). Q8_0 runs there in the low-RAM mode by itself (100 GiB of experts locked in RAM); the mix keeps every expert in RAM.
• ¹ NVIDIA FP4-Local is NVIDIA's retrained 4-bit model (quantization-aware distillation): its experts and n-gram table were trained again for FP4, so it is a slightly different model rather than a rounding of the original. Its KL to the Q8_0 model is 0.051 (the NVFP4 GPTQ and NVIDIA's NVFP4: 0.038), while its perplexity on these texts is a little below Q8_0's. Measured on 2026-10-24 on the same texts against the same reference, with the engine of this fork's 0.2.0-beta-nvfp4.3. The NVFP4 GPTQ model gives the same accuracy on that engine (93.65%), and in the same runs it decoded 180-191 tokens/s, so the two run at the same speed within the noise. NVIDIA ships ModelOpt safetensors (95 GiB) that are converted on the PC (upstream Strata's tools/nvfp4_local_pack.py prepare, about 10 minutes, 99 GiB of files); this fork's setup does not install it. Its license, the NVIDIA Open Model Agreement, rules out the censorship switch.

Quick start (Windows): from the Strata NVFP4 release, run START-HERE.bat --family qwen-nvidia-nvfp4: it checks the PC, downloads this repository and starts the model.

• Chain: Qwen/Qwen3.8-Flash-Next (BF16) → nvidia/Qwen3.8-Flash-Next-NVFP4 (NVIDIA Model Optimizer 0.46: every routed expert NVFP4, the rest BF16; revision fc694b54) → this conversion.
• NVIDIA's experts, bit for bit. Each routed expert keeps NVIDIA's 4-bit codes, FP8 block scales and FP32 global scales exactly; they are only moved into Strata's layout (checked byte for byte, below). Nothing was re-quantized.
• The quality is ModelOpt's. NVIDIA rounds each weight to the nearest NVFP4 code (round-to-nearest with MSE-calibrated scales). The fork's GPTQ quantization of the same BF16 model is another option of the same size and speed (--family qwen-nvfp4-gptq).
• The original model, censored. This is Qwen's own Flash-Next: it refuses what Qwen refuses. Its license forbids removing that, so the fork's censorship switch is not offered for this model.
• Only for Strata NVFP4. The files are Strata's runtime formats (an expert pack, GGUFs the fork reads, the MTP draft head). They do not load in transformers, vLLM or llama.cpp; for those, use NVIDIA's repository.

Files

file	size	Strata flag	what it is
pack/experts.bin	63.28 GiB	--pack pack	the routed experts, 48 layers x 512, NVIDIA's NVFP4 bits
pack/native_experts.txt	1.6 KB		the experts' layout
pack/dense.bin, pack/index.txt	1.43 GiB		weights the engine keeps in its own form: hyper-connections, routers, PLE key/value, indexer, DeltaNet gates
pack/tokenizer/	10.1 MB		the server's tokenizer and chat template
qwen-nvidia-nvfp4-dense.gguf	5.58 GiB	--native, --native-dense-gguf	the model's metadata and dense projections (Q8_0 / BF16) and head, without the routed experts
ple-fp8.gguf	47.68 GiB	--ple-gguf	the per-layer n-gram (PLE) table in FP8 E4M3, NVIDIA's bytes (= Qwen-FP8's), read from the SSD
token-embd-bf16.gguf	1.18 GiB	--embd-gguf	the token embedding in BF16 as shipped
mtp/	0.77 GiB	--mtp mtp	Qwen's MTP draft head (Q2_0 experts), for speculative decoding
LICENSE-NVIDIA-Open-Model-License.txt, NOTICE, LICENSE-Qwen			the licenses and the attribution notice
SHA256SUMS, SIZES.txt			the files' SHA-256 and sizes

What the conversion did

NVIDIA's checkpoint: the routed experts of the main model are W4A4 NVFP4 (U8 packed codes, F8_E4M3 scales per 16 values, an F32 weight_scale_2 per expert and projection); attention, DeltaNet, the shared experts, routers, hyper-connections, the embedding and the head are BF16 as Qwen ships them; the PLE n-gram table is per-tensor FP8 and the MTP head's routed experts 128x128-block FP8, both copied by NVIDIA from Qwen/Qwen3.8-Flash-Next-FP8. The vision tower is BF16 and unquantized.

• Experts (pack/experts.bin): tools/nvfp4_convert.py (llama.cpp's converter at the fork's pinned commit) repacks ModelOpt NVFP4 into ggml's block_nvfp4 by moving bits, and tools/iq_pack.py writes each expert as one blob with a 16-byte tail of its three global scales. tools/nvfp4_verify.py --scan-all: all 73,728 block-scale tensors (7.03 GiB) have no sign bit and no NaN, and dequantized experts agree bit for bit; 30 blobs rebuilt straight from the checkpoint (layers 0, 1, 17, 23, 30, 47) equal experts.bin byte for byte. NVIDIA's input_scales are not used: Strata quantizes activations itself.
• Dense weights (qwen-nvidia-nvfp4-dense.gguf, pack/dense.bin): the same policy as every Strata NVFP4 model - the projections the engine serves natively (attention, DeltaNet out, shared experts, head, embedding table in the GGUF) as Q8_0, the small ones its kernels read in BF16 as BF16. The GGUF has the same 1,511 tensors, types and shapes as the fork's OrcaRouter conversion; the 144 routed-expert tensors (63.3 GiB) are left out of the uploaded file, as next to pack/experts.bin the engine never reads them (the other 1,367 tensors and the metadata are byte-identical to the full GGUF).
• PLE table (ple-fp8.gguf): tools/ple_fp8_pack.py copies NVIDIA's 128 FP8 shards and scale without decoding. They are Qwen-FP8's bytes (checked by range reads of Qwen-FP8 on three shards). Note: the ple-fp8.gguf of the fork's OrcaRouter and Huihui repositories is a different FP8 rounding of the same BF16 table (2.4% of the bytes differ, 1.4% RMS); this repository ships Qwen's.
• Embedding (token-embd-bf16.gguf): BF16 embed_tokens, byte-identical in NVIDIA's checkpoint and Qwen's (SHA-256 of the tensor 62288a3f...).
• MTP head (mtp/): Strata's draft head runs Q2_0 experts, packed by tools/mtp_pack.py / tools/mtp_rt.py from Qwen's original BF16 head. NVIDIA's head is the same: its 29 non-expert tensors are byte-identical to Qwen's BF16 ones, and its routed experts are Qwen-FP8's FP8 copy of the same weights. draft_vocab.bin is the fork's.
• Not included: the image encoder. Setup downloads Qwen's (ISTA-DASLab's mmproj-Qwen3.8-Flash-Next-BF16.gguf, the original vision tower, which NVIDIA left in BF16).

Checks

Strata NVFP4 0.1.41-nvfp4.3 on an RTX 5090 (32 GB) with 128 GB of RAM, the arguments under Run it (--spec 6 --spec-min-p 0.7, expert cache auto), the pack on a PCIe 5 NVMe drive.

• It answers. "Explain in detail how a hash table works ... at least 300 words" (no thinking): a correct 459-token answer that ends by itself (EOS), 172.4 tok/s, 2.34 tokens per speculative round (MTP drafts accepted 0.718). A 2,099-token prompt (a CUDA source file cut off mid-line, no question): prefill 2,704 tok/s; with --stop-eos the model completes the cut identifier (q2[l -> q2[l0 /, as the file's pattern goes) and ends the turn after 4 tokens.

• Decode speed equals the fork's other NVFP4 models. Interleaved pairs on the hash-table question, each answer to its end, against the fork's OrcaRouter all-NVFP4 GPTQ pack (same format and size, its own GGUF, embedding and MTP head):

  pair	this repo: ms per round	tok/s	OrcaRouter: ms per round	tok/s
  1	13.37	172.4	13.53	172.8
  2	13.60	173.1	13.53	176.9
  3	15.79	149.6	14.32	162.7

  The third pair ran slower for both models: the PC was downloading 360 GB at the time, and the card is also the desktop's GPU. That pair is noise, not a difference between the packs.

• No reasoning loop on a long agentic request. A captured Claude Code request (72K tokens of conversation, 24 tools, thinking on, greedy decoding) that sent another abliterated Flash-Next build into an endless thinking loop in 5 of 5 runs, replayed 5 times through the fork's server with a fresh engine each time: 5 of 5 OK, thinking closed after 856, 1,280, 2,181, 950 and 1,384 tokens, each run then answering with a tool call (OrcaRouter's build: 5 of 5 OK as well).

Run it

Build or download Strata NVFP4 (Windows, NVIDIA driver 580+; see its README for the requirements: RAM, pagefile, large pages). Then:

hf download Maximilian228/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Strata --local-dir models\qwen-nvidia

One-shot (prompt.txt holds token ids):

strata.exe --pack models\qwen-nvidia\pack ^
  --native models\qwen-nvidia\qwen-nvidia-nvfp4-dense.gguf --native-dense-gguf models\qwen-nvidia\qwen-nvidia-nvfp4-dense.gguf ^
  --ple-gguf models\qwen-nvidia\ple-fp8.gguf --embd-gguf models\qwen-nvidia\token-embd-bf16.gguf ^
  --mtp models\qwen-nvidia\mtp --spec 6 --spec-min-p 0.7 --prefill auto ^
  --expert-profile data\expert-profile.bin --expert-cache auto ^
  --max-context 262144 --kv int8 --tokens-file prompt.txt --max-new 512 --stop-eos

--max-context 262144 fits a 32 GB card; with less VRAM lower it (the fork's README and setup use these):

card's VRAM	--max-context
32 GB	262144
24 GB	131072
16 GB	65536
12 GB	32768

data\expert-profile.bin ships with the engine. As a server (OpenAI and Anthropic APIs), put the same arguments in the release's config\strata-nvfp4.json ("args") and point "tokenizer" at models/qwen-nvidia/pack/tokenizer. Pictures need the image encoder (mmproj-Qwen3.8-Flash-Next-BF16.gguf from ISTA-DASLab, which setup downloads); point the config's "vision" entry's "model" at qwen-nvidia-nvfp4-dense.gguf (the encoder reads only its vocabulary).

Reproduce

With the fork's tools (its .venv, PYTHONPATH=third_party\llama.cpp\gguf-py):

hf download nvidia/Qwen3.8-Flash-Next-NVFP4 --revision fc694b54fb0174e0913e6adf86691ef85a4ead47 --local-dir models\nvidia-qwen-nvfp4
.venv\Scripts\python tools\nvfp4_convert.py --model models\nvidia-qwen-nvfp4 --outfile models\qwen-nvidia-nvfp4.gguf
.venv\Scripts\python tools\iq_pack.py --gguf models\qwen-nvidia-nvfp4.gguf --out pack
.venv\Scripts\python tools\ple_fp8_pack.py --model models\nvidia-qwen-nvfp4 --out ple-fp8.gguf
.venv\Scripts\python tools\embd_bf16_pack.py --model models\nvidia-qwen-nvfp4 --out token-embd-bf16.gguf
.venv\Scripts\python tools\mtp_fetch.py fetch --out mtp-qwen          (Qwen's BF16 MTP head, range-read)
.venv\Scripts\python tools\mtp_pack.py --src mtp-qwen --experts q2_0 --out mtp-qwen\mtp-q2_0.gguf
.venv\Scripts\python tools\mtp_rt.py --gguf mtp-qwen\mtp-q2_0.gguf --out mtp
copy data\draft_vocab.bin mtp\

The converter takes ~5 minutes and little RAM (free RAM never fell below 38 GiB on a 128 GB PC while it ran beside a download); the dense-only GGUF is the full one without blk.*.ffn_{gate,up,down}_exps.weight. The strata.ple.source and strata.embd.source metadata strings of the uploaded files name where their bytes come from; files made with the commands above differ from them only there.

License

• NVIDIA Open Model License (Last Modified: October 24, 2025), as NVIDIA ships its checkpoint. Licensed by NVIDIA Corporation under the NVIDIA Open Model License (see NOTICE). Use must also follow NVIDIA's Trustworthy AI terms. This conversion changes no safety behaviour of the model.
• The base model's Qwen Community License 1.0 (Qwen3.8-Flash-Next's LICENSE, which NVIDIA's card names as additional information): among its terms, a separate license from Qwen for commercial Model-as-a-Service or AI Work Assistant businesses.
• Strata NVFP4 itself is MIT. "NVIDIA" and "Qwen" name where the weights come from; this repository is not made or endorsed by NVIDIA or Qwen.