smalinin/DeepSeek-V4.1-Flash-GGUF

DeepSeek-V4.1-Flash GGUF

GGUF conversions and mixed-precision quantizations of deepseek-ai/DeepSeek-V4.1-Flash for the experimental DeepSeek-V4.1 runtime in smalinin/llama.cpp, branch my_build_deepseek41. [! ONLY_for_CUDA ]

This is DeepSeek-V4.1-Flash (DeepseekV41ForCausalLM), not DeepSeek-V4-Flash-0731. The architecture includes sparse attention, hyper-connections, MoE layers, and two very large n-gram Engram lookup tables.

These files require the linked my_build_deepseek41 branch. Compatibility with stock upstream llama.cpp is not claimed.

Available and planned files

Variant	Repository state	First shard	Effective whole-model BPW	Total size
Q2_K backbone + Q5_K Engram	Available	Q2_K-Q5/DeepSeek-V4.1-Flash-EngramQ5-Q2_K-00001-of-00010.gguf	3.58	335.382 GB
IQ2_XXS gate/up + Q2_K down + Q8_0 Engram (antirez-like)	Available	IQ2_XXS_GU-Q2_K_Down-Q8/DeepSeek-V4.1-Flash-EngramQ8-IQ2_XXS_GU-Q2_K_Down-00001-of-00010.gguf	3.97	371.438 GB
Protected Q2_K + Q8_0 Engram	Available	Q2_K_Protected-Q8/DeepSeek-V4.1-Flash-EngramQ8-Q2_K_Protected-00001-of-00010.gguf	4.42	413.390 GB
IQ3_XS backbone + Q8_0 Engram	Available	IQ3_XS-Q8/DeepSeek-V4.1-Flash-EngramQ8-IQ3_XS-00001-of-00010.gguf	4.64	434.024 GB
MXFP4 conversion + Q8_0 Engram	Available	MXFP4/DeepSeek-V4.1-Flash-MXFP4-00001-of-00010.gguf	5.43	507.954 GB

Download every shard of the selected variant and pass only shard 00001-of-00010 to llama-server; the remaining shards are discovered automatically.

Precision and size breakdown

Variant	Primary backbone format	Nominal format BPW	Backbone and other data	Engram format	Engram BPW	Two Engram tables	Total	Effective total BPW
Q2_K + Q5_K Engram	Q2_K	2.625	200.210 GB	Q5_K	5.50	135.172 GB	335.382 GB	3.58
IQ2_XXS gate/up + Q2_K down + Q8_0 Engram	IQ2_XXS / Q2_K hybrid	2.06 / 2.625	162.536 GB	Q8_0	8.50	208.902 GB	371.438 GB	3.97
Protected Q2_K + Q8_0 Engram	Q2_K / Q3_K protected hybrid	2.625 / 3.4375	204.488 GB	Q8_0	8.50	208.902 GB	413.390 GB	4.42
IQ3_XS + Q8_0 Engram	IQ3_XS	3.30	225.122 GB	Q8_0	8.50	208.902 GB	434.024 GB	4.64
MXFP4 + Q8_0 Engram	MXFP4	4.00	299.052 GB	Q8_0	8.50	208.902 GB	507.954 GB	5.43

The size split is exact for the completed local ten-shard artifacts. “Two Engram tables” means only blk.1.engram_embd.weight and blk.14.engram_embd.weight. “Backbone and other data” is the total shard size minus those two tensor payloads, so it also includes embeddings, output and auxiliary tensors, metadata, alignment, and per-shard overhead.

The nominal BPW column describes the named storage format, not every tensor in the file. These are deliberately hybrid models: numerically sensitive tensors remain in higher precision, and K-quant presets may select different types for different matrices. Consequently, effective whole-model BPW is the useful number for comparing final download sizes.

The MXFP4 conversion is also mixed precision, not a pure 4-bit model and not a Q8-equivalent model. Its 1,046 tensors are stored as 120 MXFP4, 332 Q8_0, 65 BF16, and 529 F32 tensors. Both Engram tables are Q8_0.

The antirez-like variant is a deliberately asymmetric MoE quant. Routed gate and up expert matrices use IQ2_XXS, while routed down matrices use Q2_K. Attention projections, shared-expert matrices, the output head, and Engram WKV use Q8_0; token embeddings and Engram Q/K remain BF16. The two large Engram embedding tables are Q8_0. The quant was produced with the same published 262,144-token importance matrix used for the other imatrix variants.

Protected Q2_K: faster than our earlier antirez-like CUDA release

The new Q2_K_Protected-Q8 release revises our own earlier IQ2_XXS_GU-Q2_K_Down-Q8 recipe for this CUDA llama.cpp runtime. That earlier recipe was inspired by antirez's Q2 design; neither release is a requantization of antirez's GGUF. It was quantized from our MXFP4 conversion using the published 262,144-token imatrix. Antirez reports an 8,192-token imatrix for his DwarfStar release; a larger calibration set is useful coverage, but does not by itself prove better output quality.

The original antirez release is a design reference, not the speed baseline for the comparison below:

Design choice	Original antirez Q2	This Protected Q2_K release
Routed gate/up	IQ2_XXS	Q2_K
Routed down	Q2_K	Q3_K
Attention, shared experts, output	Q8	Q5_K/Q6_K mixed policy
Engram tables	Native FP8	Q8_0
Activation-imatrix corpus	8,192 tokens	262,144 tokens
Whole Q2 download	340.60 GiB	385.00 GiB

The extra bits on routed experts and larger calibration corpus are intended to improve this CUDA-oriented precision/speed balance. The lower-precision dense tensors, different Engram representation and different checkpoint revisions mean the table is not proof of universally better quality.

Compared with our earlier IQ2_XXS/Q2_K antirez-like variant, this recipe spends more precision on the routed experts: gate and up use Q2_K instead of IQ2_XXS, and down uses Q3_K instead of Q2_K. The two huge Engram tables remain Q8_0. Token embeddings, protected indexer K weights, Engram Q/K and expert routers remain BF16; mHC tensors remain F32. Indexer Q projections use Q8_0. To balance the larger expert matrices, attention projections and the output head use Q6_K, shared-expert gate/up use Q5_K, shared-expert down and Engram WKV use Q6_K. Those dense tensors were Q8_0 in the previous antirez-like recipe, so overall quality superiority has not been established without a matched evaluation.

On the maintainer's six-GPU CUDA setup, Protected Q2_K generates faster than our earlier IQ2_XXS_GU-Q2_K_Down-Q8 release in this llama.cpp runtime. This is a comparison between our two GGUF variants, not against antirez's separate DwarfStar/Metal release. Using Q2_K/Q3_K CUDA kernels for routed experts is consistent with the observed speedup, but its individual contribution has not been isolated, and a matched public A/B table is not yet available. One target-only smoke test of Protected Q2_K measured tg32 = 35.70 tok/s after a 512-token prompt and pp60000 = 272.07 tok/s with ubatch=256, six GPUs and a 64K fit context. These are separate workloads, not a combined request or a quality score. The extra precision costs space: 413.390 GB versus 371.438 GB for our IQ2_XXS antirez-like variant. The two Q8_0 Engram tables are mmap/lazy host-backed with the recommended launch options, not loaded wholesale into VRAM.

Which variant should I use?

• Q2_K + Q5_K Engram is the smallest release. It is the practical choice when storage, RAM, or aggregate GPU memory is the limiting factor.
• IQ2_XXS gate/up + Q2_K down + Q8_0 Engram is the antirez-like compact hybrid. It saves space in the routed experts while protecting the dense path and both Engram tables. It is smaller than IQ3_XS + Q8_0, but IQ2_XXS is an aggressive expert quantization and is not guaranteed to be faster than Q2_K on every CUDA backend.
• Protected Q2_K + Q8_0 Engram generates faster than our earlier IQ2_XXS_GU-Q2_K_Down-Q8 variant on the maintainer's CUDA system. It protects routed expert precision and keeps Q8_0 Engram, at the cost of a larger download and lower precision for some dense tensors. Choose it when CUDA generation speed and the routed expert precision budget matter more than the smallest file size.
• IQ3_XS + Q8_0 Engram is the middle tier. It uses the calibration importance matrix and preserves both Engram tables in Q8_0.
• MXFP4 + Q8_0 Engram is the conversion source and highest-precision tier in this repository. It avoids a second backbone quantization pass, but is substantially larger.

Lower BPW can increase repetition or otherwise reduce output quality. Evaluate the variants on your own prompts; the format name alone is not a quality guarantee.

Reproducible conversion

The source checkpoint used here is:

deepseek-ai/DeepSeek-V4.1-Flash@dba1be0a40aa45a94ad051997016db3960a90277

The accepted MXFP4 artifact was converted with the DeepSeek-V4.1 converter at smalinin/llama.cpp commit 4c7025b52339fec48d8e8bdfcea905db80130b9c. The examples below use local paths only as placeholders; change them for your system.

0. Build the required branch

git clone --branch my_build_deepseek41 \
  https://github.com/smalinin/llama.cpp.git
cd llama.cpp

cmake -S . -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j \
  --target llama-server llama-quantize llama-imatrix llama-gguf-split

python3 -m pip install -r requirements.txt

Use the CUDA/toolchain settings appropriate for your system. The commands in the following sections assume this checkout is assigned to LLAMA_CPP.

1. Create the MXFP4 GGUF

Install the converter requirements from the linked branch first, then run:

LLAMA_CPP=/path/to/llama.cpp
HF_MODEL=/path/to/DeepSeek-V4.1-Flash
OUT=/path/to/output

mkdir -p "$OUT/tmp"

TMPDIR="$OUT/tmp" \
/usr/bin/time -v python3 "$LLAMA_CPP/convert_hf_to_gguf.py" \
  "$HF_MODEL" \
  --outtype auto \
  --use-temp-file \
  --outfile "$OUT/DeepSeek-V4.1-Flash-MXFP4.gguf"

--outtype auto invokes the architecture-specific precision policy: supported official block-FP4 expert weights are written as MXFP4, while tensors requiring another representation are kept as Q8_0, BF16, or F32.

The conversion needs substantial temporary disk space and RAM. The reference run had approximately 193 GiB peak RSS and took about 2 h 59 min on the build machine.

2. Split the MXFP4 file

"$LLAMA_CPP/build/bin/llama-gguf-split" \
  --split \
  --split-max-size 48G \
  "$OUT/DeepSeek-V4.1-Flash-MXFP4.gguf" \
  "$OUT/DeepSeek-V4.1-Flash-MXFP4"

This produced ten shards. A single GGUF tensor cannot be split across files, so a shard containing one of the approximately 104.45 GB Q8_0 Engram tables can exceed the requested 48G limit.

Importance matrix

All imatrix quants use the same 262,144-token calibration corpus and the published imatrix-dsv41-262144.gguf. The accepted run used 512 chunks:

"$LLAMA_CPP/build/bin/llama-imatrix" \
  --model "$OUT/DeepSeek-V4.1-Flash-MXFP4-00001-of-00010.gguf" \
  --file /path/to/calibration.txt \
  --output "$OUT/imatrix-dsv41-262144.gguf" \
  --output-format gguf \
  --ctx-size 512 \
  --batch-size 512 \
  --ubatch-size 128 \
  --parallel 1 \
  --chunks 512 \
  --output-frequency 512 \
  --save-frequency 128 \
  --no-ppl \
  --parse-special \
  --no-escape \
  --n-gpu-layers auto \
  --split-mode layer \
  --fit on \
  --fit-ctx 512 \
  --fit-target 2048 \
  --load-mode mmap \
  --lazy-mode auto \
  --n-cpu-moe 0 \
  --flash-attn on

Quantization commands

The quantizer reads the complete MXFP4 split set through its first shard. --keep-split preserves the shard layout. The output path below is a base GGUF name; the quantizer creates numbered shards.

Q2_K backbone + Q5_K Engram

"$LLAMA_CPP/build/bin/llama-quantize" \
  --allow-requantize \
  --keep-split \
  --max-buffer-size 8192 \
  --imatrix "$OUT/imatrix-dsv41-262144.gguf" \
  --tensor-type 'engram_embd\.weight=q5_k' \
  "$OUT/DeepSeek-V4.1-Flash-MXFP4-00001-of-00010.gguf" \
  "$OUT/DeepSeek-V4.1-Flash-EngramQ5-Q2_K.gguf" \
  Q2_K 20

IQ3_XS backbone + Q8_0 Engram

"$LLAMA_CPP/build/bin/llama-quantize" \
  --allow-requantize \
  --keep-split \
  --max-buffer-size 8192 \
  --imatrix "$OUT/imatrix-dsv41-262144.gguf" \
  --token-embedding-type q5_k \
  --tensor-type 'engram_embd\.weight=q8_0' \
  "$OUT/DeepSeek-V4.1-Flash-MXFP4-00001-of-00010.gguf" \
  "$OUT/DeepSeek-V4.1-Flash-IQ3_XS-Q8_0_Engram-imatrix.gguf" \
  IQ3_XS 20

IQ2_XXS gate/up + Q2_K down + Q8_0 Engram (antirez-like)

"$LLAMA_CPP/build/bin/llama-quantize" \
  --allow-requantize \
  --keep-split \
  --max-buffer-size 8192 \
  --imatrix "$OUT/imatrix-dsv41-262144.gguf" \
  --tensor-type 'ffn_(gate|up)_exps\.weight=iq2_xxs' \
  --tensor-type 'ffn_down_exps\.weight=q2_k' \
  --tensor-type 'engram_(embd|wkv)\.weight=q8_0' \
  --tensor-type 'attn_.*\.weight=q8_0' \
  --tensor-type 'ffn_(gate|up|down)_shexp\.weight=q8_0' \
  --tensor-type '^output\.weight=q8_0' \
  --tensor-type '^token_embd\.weight=bf16' \
  "$OUT/DeepSeek-V4.1-Flash-MXFP4-00001-of-00010.gguf" \
  "$OUT/DeepSeek-V4.1-Flash-IQ2_XXS_GU-Q2_K_Down-Q8_0_Engram-imatrix.gguf" \
  Q2_K 20

Protected Q2_K + Q8_0 Engram (faster than our IQ2_XXS/Q2_K variant on CUDA)

"$LLAMA_CPP/build/bin/llama-quantize" \
  --allow-requantize \
  --keep-split \
  --max-buffer-size 8192 \
  --imatrix "$OUT/imatrix-dsv41-262144.gguf" \
  --tensor-type '^blk\.[0-9]+\.ffn_(gate|up)_exps\.weight=q2_k' \
  --tensor-type '^blk\.[0-9]+\.ffn_down_exps\.weight=q3_k' \
  --tensor-type '^blk\.[0-9]+\.indexer\.attn_(k|q_b)\.weight=q8_0' \
  --tensor-type '^blk\.[0-9]+\.attn_(kv|output_a|output_b|q_a|q_b)\.weight=q6_k' \
  --tensor-type '^blk\.[0-9]+\.ffn_(gate|up)_shexp\.weight=q5_k' \
  --tensor-type '^blk\.[0-9]+\.ffn_down_shexp\.weight=q6_k' \
  --tensor-type '^output\.weight=q6_k' \
  --tensor-type '^token_embd\.weight=bf16' \
  --tensor-type '^blk\.[0-9]+\.engram_embd\.weight=q8_0' \
  --tensor-type '^blk\.[0-9]+\.engram_wkv\.weight=q6_k' \
  --tensor-type '^blk\.[0-9]+\.engram_(q|k)\.weight=bf16' \
  --tensor-type '^blk\.[0-9]+\.ffn_gate_inp\.weight=bf16' \
  --tensor-type '^blk\.[0-9]+\.hc_.*\.weight=f32' \
  "$OUT/DeepSeek-V4.1-Flash-MXFP4-00001-of-00010.gguf" \
  "$OUT/DeepSeek-V4.1-Flash-Protected-Q2_K-Q8_0_Engram-imatrix.gguf" \
  Q2_K 20

The quantizer keeps indexer.attn_k.weight in BF16 despite the requested Q8_0 override; the audited dry run verified this protection before writing the ten shards. The resulting GGUF is 4.42 effective BPW.

The final positional argument is the worker-thread count; tune it for your machine. --allow-requantize is required because the source GGUF is already a mixed-precision conversion.

Running with llama-server

The following is the tested six-GPU profile for the Q2_K + Q5_K Engram release. Adjust the GPU order, paths, thread counts, and fit target for your hardware. For Protected Q2_K, use the same target-only profile and replace --model with Q2_K_Protected-Q8/DeepSeek-V4.1-Flash-Protected-Q2_K-Q8_0_Engram-imatrix-00001-of-00010.gguf.

/path/to/llama.cpp/build/bin/llama-server \
  --model /path/to/Q2_K-Q5/DeepSeek-V4.1-Flash-EngramQ5-Q2_K-00001-of-00010.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 64000 \
  --batch-size 2048 \
  --ubatch-size 256 \
  --parallel 1 \
  --threads 4 \
  --n-gpu-layers auto \
  --split-mode layer \
  --fit on \
  --fit-ctx 64000 \
  --fit-target 1024 \
  --load-mode mmap \
  --lazy-mode auto \
  --flash-attn on \
  --n-cpu-moe 0 \
  --no-warmup \
  --no-context-shift \
  --jinja \
  --chat-template-kwargs '{"reasoning_effort":80,"enable_thinking":true}' \
  --reasoning-format deepseek \
  --no-reasoning-preserve \
  --no-prefill-assistant

Important launch parameters

Parameter	Purpose
--ctx-size 64000	Allocates a 64K-token context for the single server slot. A larger capacity increases cache and graph memory.
--batch-size 2048	Maximum logical prompt batch. It mainly affects prompt processing.
--ubatch-size 256	Physical micro-batch used to execute prompt work. Reduce it if graph memory is too high.
--parallel 1	Uses one server slot. With multiple slots, the configured context is divided among them.
--n-gpu-layers auto --split-mode layer	Lets the custom runtime place layers across the visible GPUs.
--fit on --fit-ctx 64000 --fit-target 1024	Runs the placement fitter for the requested context while targeting approximately 2 GiB of free VRAM per GPU.
--load-mode mmap --lazy-mode auto	Memory-maps the very large GGUF and allows the runtime to keep suitable tensors, notably Engram tables, lazily backed by host storage.
--flash-attn on	Enables the supported Flash Attention path.
--n-cpu-moe 0	Requests GPU placement for all MoE layers. Change this only when the model does not fit.
--no-warmup	Skips startup warm-up. Remove this option if you prefer to pay warm-up cost before the first real request.
--no-context-shift	Disables rolling context reuse. A request that exceeds the configured context must be rejected or shortened instead of silently shifting it.
--jinja --chat-template-file ...	Uses the supplied DeepSeek-V4.1 chat template. The template file is also included in this repository.
reasoning_effort	Template value from 1 to 100. Higher values request more thorough and usually longer reasoning; they do not change model weights.
--reasoning-format deepseek	Parses DeepSeek reasoning into the server's reasoning field.
--no-reasoning-preserve	Does not preserve extracted reasoning when assistant history is reconstructed.
--no-prefill-assistant	Disables assistant-message prefill.

The two Engram tables are intentionally compatible with mmap/lazy host-backed operation; total GGUF size is therefore not the same as required VRAM. Actual RAM and VRAM requirements still depend on context length, batch sizes, GPU topology, and placement selected by the fitter.

Engram tables and VRAM

With the recommended --load-mode mmap --lazy-mode auto configuration, the two giant engram_embd.weight lookup tables are not loaded into VRAM. The DeepSeek-V4.1 runtime marks them for lazy host-side row access: their GGUF storage is memory-mapped and only the requested rows are gathered for model execution. Consequently, the approximately 135 GB Q5_K pair or 209 GB Q8_0 pair must not be added directly to the model's VRAM requirement.

Demanded mmap pages can become resident in the operating system's host RAM page cache and may later be evicted. This is different from permanently allocating the complete tables in RAM or VRAM. The smaller Engram Q/K and WKV projection tensors are ordinary model tensors and can be placed separately by the runtime/fitter.

This statement is specific to lazy mode. Disabling it with --lazy-mode off makes the Engram tables follow ordinary tensor placement and can dramatically increase RAM or VRAM requirements. Keep --load-mode mmap --lazy-mode auto unless you intentionally want to test a different placement policy.

Optional DSpark sidecar

DeepSeek-V4.1-Flash-DSpark-AUTO.gguf is an experimental three-stage MTP speculative-decoding sidecar (approximately 7.97 GB). It is not enabled in the recommended command above: on the tested system it accepted draft tokens correctly but reduced end-to-end generation speed. If you want to experiment, add the following options and measure on your own workload:

  --spec-type draft-dspark \
  --spec-draft-model /path/to/DeepSeek-V4.1-Flash-DSpark-AUTO.gguf \
  --spec-draft-n-max 5 \
  --gpu-layers-draft auto

Scope and limitations

• Text generation is supported; the checkpoint's vision tower is not mapped into these GGUF files.
• The native one-million-token training context has not been claimed as a validated deployment configuration here. The documented production profile uses a 64K context.
• Q2_K is an aggressive quantization. Repetition or quality loss at this size is a model-quantization limitation, not necessarily a runtime failure.
• Performance depends heavily on memory placement, PCIe topology, context length, page-cache state, and the number and type of GPUs. No universal tokens-per-second figure is claimed by this card.

Samples

Q2_K + Q5_K Engram

IQ3_XS + Q8_0 Engram

• used opencode, toolcall, vision

  

IQ2_XXS gate/up + Q2_K down + Q8_0 Engram (antirez-like recipe)

• used opencode, toolcall, vision

  

MXFP4 Higher-precision sample

Acknowledgements

The conversion and runtime work builds on llama.cpp, the official DeepSeek checkpoint and reference implementation, and early DeepSeek-V4.1 conversion and runtime work by vcruz305 and JigSawPT.