DeepSeek-V4.1-Flash-1017 · EXL3 2.9 bpw

TL;DR — EXL3 quantization of deepseek-ai/DeepSeek-V4.1-Flash-1017 at 2.90 bpw (196.3 GiB). On held-out text its perplexity is +1.8% (code) / +0.4% (prose) above the unquantized model. Tensor names, dtypes and per-tensor bitrates match Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw tensor for tensor, so the MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks serving kit's bundled K map describes it unchanged. The two Engram tables are not in this repo: take them from the 1017 base repo, because they changed in this release.

Tensor groups

Every EXL3 matrix is stored as trellis (int16) + suh / svh (fp16) + mul1 (int32 codebook marker).

Perplexity

Two held-out sets of unpublished text, 32 sequences x 2048 tokens each, tokens 2-2048 of every sequence scored (65,504 tokens per set). Every row uses the same full-sequence forward (see How it was made).

Engram tables (required)

DeepSeek-V4.1 adds two Engram n-gram lookup tables (layers.1.engram.embed, layers.14.engram.embed: FP8 with E8M0 block scales, ~95 GiB each). They are used unquantized and come from the base checkpoint, like in the existing V4.1 EXL3 serving stacks. The 1017 tables are new: the shards from the older deepseek-ai/DeepSeek-V4.1-Flash must not be reused.

hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 \
  --include model-00047-of-00048.safetensors model-00048-of-00048.safetensors model.safetensors.index.json config.json \
  --local-dir ./engram-src-1017
sha256sum engram-src-1017/model-0004[78]-of-00048.safetensors

Serving

The checkpoint follows the layout of Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw, so it targets the vLLM-based MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks kit (2x DGX Spark, TP=2). The settings below come from the kit's start.sh / download.sh (commit 6f7d159). Upstream exllamav3 has no DeepseekV41 model class yet, so it cannot load this checkpoint directly.

Fresh install (on the head node, from the kit checkout):

git clone https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks
cd DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks
cp .env.example .env
cat >> .env <<'EOF'
HF_MODEL_REPO=LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw
MODEL_HOST=$PWD/model-1017-2.9bpw
EXPECTED_SHARDS=43
HF_ENGRAM_REPO=deepseek-ai/DeepSeek-V4.1-Flash-1017
ENGRAM_DIR=./engram-src-1017
EOF
./download.sh   # optional: stage both downloads first
./start.sh

.env is sourced by bash, so $PWD expands to the kit directory when the scripts are run from it.

The kit's bundled files/exl3_k_map.json already matches this checkpoint (this repo's exl3_k_map.json has the same values for every key), so KMAP_HOST can stay at its default.

Switching an existing V4.1 install — the kit reuses anything already on disk, so make sure nothing old gets picked up:

How it was made

323<｜end▁of▁sentence｜>

Sunlight contains all colors, but air molecules scatter short blue wavelengths much more than longer red ones—a process called Rayleigh scattering. This scattered blue light reaches your eyes from all directions, making the sky appear blue. At sunrise and sunset, light travels through more atmosphere, so blue is scattered away and reds and oranges dominate.<｜end▁of▁sentence｜>

We need answer classic. Need be careful: ball 0.05, bat 1.05. Explain concise.</think>The ball costs **$0.05** (5 cents).

The bat costs $1.05, so together they total $1.10.<｜end▁of▁sentence｜>

Files

Credits

• DeepSeek for deepseek-ai/DeepSeek-V4.1-Flash-1017 (MIT).
• turboderp for exllamav3 (MIT): quantizer, kernels and conversion pipeline.
• Mia-AiLab for Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw and the MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks kit, whose tensor layout and K map this release follows.

License

MIT, same as the base model.