dsx

Run DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) at full routing on hardware you can own or rent: a desktop with two RTX 3090s, or one AMD Instinct MI325X. Two runners built for this one model, each with its own quant, sharing one server (OpenAI and Anthropic compatible, tool calls, streaming, web UI).

Both use exact routing (no expert pruning, no router bias). Quality is measured on Lucebox's 1017 eval set (8 x 1024 tokens against the FP8 model's top-256 log-probs). The Engram tables (203 GB) come from the official checkpoint for both.

NVIDIA desktop

On the single-GPU box, ~40 GB of experts live on disk, which sets its speed.

# weights: the quant, plus the Engram tables and tokenizer from the official checkpoint
hf download lxe/DeepSeek-V4.1-Flash-1017-dsx-2bit --local-dir dsx-2bit
hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 --local-dir v41-1017 \
  --include model.safetensors.index.json model-00047-of-00048.safetensors model-00048-of-00048.safetensors tokenizer.json

# build (86 = RTX 30xx, 120 = RTX 50xx) and serve
CUDA_ARCHS=86 cargo build --release -p dsx --features cuda
./target/release/dsx-server --model dsx-2bit/DeepSeek-V4.1-Flash-1017-dsx-2bit-00001-of-00004.gguf \
  --engram v41-1017 --tokenizer v41-1017/tokenizer.json --host 0.0.0.0 --port 8090 \
  --ctx 131072 --gpu0-gb=-1.5 --gpu1-gb=-1.0 --arena-gb 110 --rank dsx-2bit/expert-rank.json

Placement

Experts are spread across VRAM, a RAM arena and the file, hottest first.

How it works

• Within a token, GPU0 posts each layer's expert picks to shared memory and the CPU and GPU1 answer through flags, so the whole token runs without waiting on the host.
• Prefill runs the first half of the layers on every token and the second half only on the tokens it can still affect.
• The server reuses the cached prefix across turns.

AMD MI325X

One AMD Instinct MI325X (gfx942, 256 GiB HBM3E) holds the whole model except the Engram tables, so the checkpoint's own MXFP4 experts can stay as they are for the 59% most used; the rest get a fitted 3-bit codebook.

hf download lxe/DeepSeek-V4.1-Flash-1017-dsx-mxfp4 --local-dir dsx-mxfp4
hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 --local-dir v41-1017 \
  --include config.json tokenizer.json model.safetensors.index.json model-00047-of-00048.safetensors model-00048-of-00048.safetensors
cp dsx-mxfp4/dsx-engram.json v41-1017/

cargo build --release -p dsx-amd      # ROCm 7, gfx942
DSXA_WEIGHTS=dsx-mxfp4/DeepSeek-V4.1-Flash-1017-dsx-mxfp4-00001-of-00006.gguf \
  ./target/release/dsx-amd serve --model v41-1017 --ctx 262144 --host 0.0.0.0 --port 8090

--model is the official checkpoint directory (config, tokenizer, Engram shards, plus dsx-engram.json from the quant repo). DSXA_DSPARK=3 adds the checkpoint's DSpark draft layers for greedy requests; they are read from the official mtp.* tensors (add model-00044-of-00048.safetensors .. model-00046-of-00048.safetensors to the download) and take 6.5 GB of HBM, so 1,613 cold experts go to 3-bit (KL 0.056). Measurements and design notes: FINDINGS-AMD.md.

Using the server

Both runners serve the same API on --port. Open http://localhost:8090/ for the web UI, or use any OpenAI client with base URL http://localhost:8090/v1:

curl http://localhost:8090/v1/chat/completions -H 'content-type: application/json' -d '{
  "model": "deepseek-v4.1-flash", "stream": true, "reasoning_effort": "none",
  "messages": [{"role": "user", "content": "Write a haiku about MoE routing."}] }'

Thinking is on by default; "reasoning_effort": "none" or "thinking": {"type": "disabled"} turns it off.

Clients

Any OpenAI-compatible client works. For the pi coding agent, add to ~/.pi/agent/models.json:

{ "providers": { "dsx": {
  "baseUrl": "http://localhost:8090/v1", "api": "openai-completions", "apiKey": "none", "authHeader": false,
  "compat": { "supportsDeveloperRole": false, "supportsReasoningEffort": true, "maxTokensField": "max_tokens" },
  "models": [{ "id": "deepseek-v4.1-flash", "reasoning": true, "input": ["text"], "contextWindow": 131072,
               "maxTokens": 8192, "thinkingLevelMap": { "off": "none", "high": "high" } }] } } }

pi --provider dsx --model deepseek-v4.1-flash:off

Tools

target/release/dsx (NVIDIA runner):

Repo

One Cargo workspace: cargo build --release builds core and the NVIDIA runner (CPU-only without --features cuda); -p dsx-amd builds the AMD runner (needs ROCm). Binaries land in target/release/.

The 2-bit quant uses dsx's per-expert formats (ternary, Q2_0, Q3_K per expert) and loads in the NVIDIA runner; the MXFP4 + 3-bit build loads in the AMD runner.

Credits

DeepSeek for the model (MIT) · Xolotl / Strata for the GSQ tools and V4.1 GGUF format (MIT) · Lucebox for the eval set · antirez for the calibration prompts · the web UI is llama.cpp's, by Georgi Gerganov and the ggml authors (MIT), vendored in core/webui with its license.

MIT license.