DeepSeek-V4.1-Flash-1017 · dsx 2-bit

DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) in 148 GB, built to run on two RTX 3090s and 128 GB of RAM with exact routing. Runs on dsx, a Rust + CUDA engine made for this model. For one AMD MI325X (256 GiB HBM) there is a near-lossless build: lxe/DeepSeek-V4.1-Flash-1017-dsx-mxfp4 (KL 0.049, 110-114 tok/s).

Quality

Measured on Lucebox's 1017 eval set: 8 × 1024 tokens against the FP8 model's top-256 log-probs.

Speed

Greedy, exact routing, 400 generated tokens.

Decode speed comes down to how many experts fit in VRAM + RAM. Two 24 GB GPUs + 128 GB hold all of them; one 32 GB GPU

• 128 GB leaves ~40 GB of experts to be read from disk.

Size

Run

# the quant: 4 GGUF shards, loaded as one model
hf download lxe/DeepSeek-V4.1-Flash-1017-dsx-2bit --local-dir dsx-2bit

# Engram tables + tokenizer from the official checkpoint
hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 --local-dir v41-1017 \
  --include model.safetensors.index.json model-00047-of-00048.safetensors model-00048-of-00048.safetensors tokenizer.json

# the engine (dsx's NVIDIA runner)
git clone https://github.com/lxe/dsx && cd dsx
CUDA_ARCHS=86 cargo build --release -p dsx --features cuda      # 86 = RTX 30xx, 120 = RTX 50xx

./target/release/dsx-server --model ../dsx-2bit/DeepSeek-V4.1-Flash-1017-dsx-2bit-00001-of-00004.gguf \
  --engram ../v41-1017 --tokenizer ../v41-1017/tokenizer.json --host 0.0.0.0 --port 8090 \
  --ctx 131072 --gpu0-gb=-1.5 --gpu1-gb=-1.0 --arena-gb 110 --rank ../dsx-2bit/expert-rank.json

Open http://localhost:8090/ for the web UI, or point any OpenAI-compatible client at http://localhost:8090/v1:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8090/v1", api_key="none")
r = client.chat.completions.create(model="deepseek-v4.1-flash", reasoning_effort="none",
                                   messages=[{"role": "user", "content": "Merge overlapping intervals in Python."}])
print(r.choices[0].message.content)

Thinking is on by default; "reasoning_effort": "none" turns it off.

Hardware sizing

VRAM + RAM should hold ~140 GB of experts.

Also needed: CUDA 13, Rust, an AVX2 CPU, ~350 GB of disk.

How it was made

• Experts: each of the 15,360 routed experts gets ternary (1.75 bpw), Q2_0 (2.25) or Q3_K (3.44), picked by its measured output error. One bit budget is shared across layers 2-36, so sensitive layers get up to 2.75 bpw and forgiving ones ~1.76. Average: 2.03 bpw.
• Method: GPTQ per expert, with Hessians from a layer-by-layer calibration pass through DeepSeek's reference model.py (GSQ pipeline from Xolotl).
• Calibration text: 128 × 2048 tokens of coding-agent sessions and antirez's agent and code prompts.
• Dense weights: Q8_0. Engram: unquantized, read from the official shards.

Pipeline: github.com/lxe/dsx/tree/main/quant.

Files

The earlier 2.0-bpw-per-layer build (KL 0.462) is at revision v1.

Credits

DeepSeek for the model (MIT) · Xolotl / Strata for the GSQ tools and V4.1 GGUF format · Lucebox for the eval set · antirez for the calibration prompts · llama.cpp (Georgi Gerganov and the ggml authors, MIT) for the web UI.