DeepSeek V4 Flash Vision-Exp DSpark on 2x DGX Spark

_{by Mia'a AI Lab}

Two-node DGX Spark recipe for deepseek-ai/DeepSeek-V4-Flash-Vision-Exp: vLLM TP=2, DSpark speculative decoding, 1M-token ceiling, nvfp4_ds_mla KV. Native image support is a startup hotfix on the Anemll 0.1.1 runtime (ViT + Aligner from the Vision-Exp checkpoint, OpenAI image_url / <image>path</image>). There is no video encoder in the official weights; GIF is a still frame. The old Qwen3-VL sidecar / MCP path is removed.

Default image: ghcr.io/anemll/dspark-vllm-gx10:0.1.1

Numbers: results/RESULTS-2026-08-14.md (dated tables, method, historical lanes). Checkpoint / encoder: docs/DEEPSEEK_V4_FLASH_0731.md.

---

Quick start

Run everything from the head node. You need two DGX Sparks, RoCE/NCCL working, and the same image on both. By default each node has its own HuggingFace checkpoint (prepare copies onto the worker). Set DSPARK_WORKER_HF_NFS=1 to keep weights only on the head: the worker mounts that cache over NFSv4 on the ConnectX link (same pattern as Qwen3.8-Flash-vLLM).

1. Env

   cp .env.dspark.example .env.dspark
   

   Set at least: WORKER_HOST, MASTER_ADDR, NCCL_IB_HCA, NCCL_SOCKET_IFNAME (and matching TP_ / GLOO_ IF names), VLLM_HOST_IP, WORKER_VLLM_HOST_IP, HF_CACHE. WORKER_HF_CACHE is the worker checkpoint path. When DSPARK_WORKER_HF_NFS=1, it is the worker's local JIT overlay on the NFS mount. If the worker checkout is not the same path, set WORKER_DIR / WORKER_SCRIPT_DIR.

   Example fabric (edit f0 vs f1 and GID for your ring):

   WORKER_HOST=10.0.0.2
   MASTER_ADDR=10.0.0.1
   VLLM_HOST_IP=10.0.0.1
   WORKER_VLLM_HOST_IP=10.0.0.2
   NCCL_IB_HCA=rocep1s0f1
   NCCL_SOCKET_IFNAME=enp1s0f1np1
   DSPARK_VLLM_IMAGE=ghcr.io/anemll/dspark-vllm-gx10:0.1.1
   

   Leave serving knobs at the defaults unless you mean to change them. Meaningful on/off flags (ABLITERATED, thinking, hotfixes) are listed under .env.dspark switches.

2. Image on both nodes

   docker pull ghcr.io/anemll/dspark-vllm-gx10:0.1.1
   

   Repeat on the worker (or pull there via ssh). Start refuses to launch if either node is missing the image.

3. Weights on the head

   ./prepare-dspark-model-cache.sh --official
   

   Use --abliterated or --yes (reads ABLITERATED from .env.dspark). Abliteration is gated (HF_TOKEN): agree on the Keys Hub repo, then prepare downloads the 18 KiB direction — not the 157 GiB checkpoint. Prepare forces HF online even if HF_HUB_OFFLINE=1, then you can serve offline. Default DSPARK_WORKER_HF_NFS=0 also downloads onto the worker. After the cache is complete, keep HF_HUB_OFFLINE=1. See Worker weights over NFS to skip the second copy.

4. Optional CPU gates (no GPU; will not measure tok/s)

   bash scripts/ci-validate.sh
   

5. Start (worker first, then head)

   ./start-deepseek-v4-flash-dspark.sh
   

   One-shot bind override: ./start-deepseek-v4-flash-dspark.sh --host 0.0.0.0 --port 9000. After a reboot, dockerd may already have restored the ranks (restart: unless-stopped); start then exits 3 (already running), not 1. That is expected — do not ./stop unless you want a cold start. systemd: SuccessExitStatus=3.

   Optional three Sparks (TP=3) is a separate launcher so .env cannot flip the 2-node path: ./start-tp3.sh (needs WORKER2_HOST; see Optional: three Sparks (TP=3) and docs/TP3.md).

6. Check it is up

   curl -fsS http://127.0.0.1:8888/v1/models
   ./smoke-deepseek-v4-flash-dspark.sh
   ./status-deepseek-v4-flash-dspark.sh
   

   Expect "id": "deepseek-v4-flash-vision-exp" and "max_model_len": 1048576. Boot log (trust the live numbers):

   Available KV cache memory: 17.04 GiB
   GPU KV cache size: 2,331,430 tokens
   Maximum concurrency for 1,048,576 tokens per request: 2.22x
   

API: http://HEAD_NODE_IP:8888/v1 (VLLM_HOST=0.0.0.0 by default). Head-only tests: VLLM_HOST=127.0.0.1.

Day-to-day: ./status-…, ./logs-…, ./stop-…. Disable earlyoom on both hosts or it can kill vLLM under deep-context load.

---

Default profile

start-*.sh exports GPU_MEMORY_UTILIZATION from GPU_MEMORY_UTILIZATION_TEXT. Do not set GPU_MEMORY_UTILIZATION by hand.

max_model_len and max_num_seqs are ceilings, not reservations. The limit is sum(live tokens) ≤ KV pool. Six normal agent turns fit; six simultaneous full-1M requests do not. See How the KV cache works.

Long coding / big prompts (optional, still 1M ceiling):

MAX_NUM_SEQS=4
MAX_NUM_BATCHED_TOKENS=16384
GPU_MEMORY_UTILIZATION_TEXT=0.87

---

.env.dspark switches

Copy .env.dspark.example → .env.dspark. Start syncs it to the worker. Restart both ranks after a flip (./stop-… then ./start-…). Do not set DSPARK_MODEL or GPU_MEMORY_UTILIZATION by hand.

NCCL/RoCE, CUDA arch, and compile knobs stay in the example file — they are cluster wiring, not product switches. Full Anemll vs Stage-C matrix: docs/ENVS.md.

Weights

Flip ABLITERATED like this:

# in .env.dspark
ABLITERATED=1

./prepare-dspark-model-cache.sh --yes    # or --abliterated / --official
./stop-deepseek-v4-flash-dspark.sh
./start-deepseek-v4-flash-dspark.sh

--official writes ABLITERATED=0; --abliterated writes 1.

Worker weights over NFS (optional)

Default is off: DSPARK_WORKER_HF_NFS=0. prepare downloads the checkpoint onto the worker as well.

Set DSPARK_WORKER_HF_NFS=1 in .env.dspark to skip that second Hub download. Start then exports the head HF_CACHE via NFSv4 on NCCL_SOCKET_IFNAME (ConnectX). A live exporter on that address is reused (for example Qwen's vllm-fn-nfs); otherwise start brings up dspark-nfs. The worker Docker volume dspark-hf mounts the share read-only. Triton, TileLang, vLLM, FlashInfer, CuTe, and NCCL-FR caches stay on the worker host as overlays under WORKER_HF_CACHE. ./stop-deepseek-v4-flash-dspark.sh --nfs tears down only dspark-nfs, not Qwen's share.

DSPARK_WORKER_HF_NFS=1
# NFS_SERVER_IP=10.0.22.1   # optional; default is IPv4 on NCCL_SOCKET_IFNAME

If official Vision-Exp is already cached, overlay the 26 edited shards (~87 GiB) instead of re-fetching the full ~157 GiB dump:

python3 scripts/overlay-vision-exp-ablit-cache.py

The 0731 abliterated freeze stays on branch 0731-ablit.

Images: OpenAI image_url (JPEG/PNG/GIF/WebP; GIF is still-frame). No video. Images belong in user messages only. A structured image / image_url part (or the raw <｜deepseek_image｜> token) on system, assistant, tool, or function returns HTTP 400. Tool/function text that quotes <image> tags is allowed; putting a vision-tool result on role: "tool" as image_url is not. That 400 is not retried by typical OpenAI clients, and the rejected turn stays in history, so later messages keep failing until that tool message is dropped (issue #178). Put screenshots on a user turn instead. Default cap 8 images (LIMIT_MM_PER_PROMPT is JSON {"image":8}; image=8 is converted). Example:

{"model":"deepseek-v4-flash-vision-exp","messages":[{"role":"user","content":[
  {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,..."}},
  {"type":"text","text":"What is in this picture?"}
]}]}

Update the pin like this:

# in .env.dspark
DSPARK_REVISION=<commit>

./prepare-dspark-model-cache.sh --yes
./stop-deepseek-v4-flash-dspark.sh
./start-deepseek-v4-flash-dspark.sh

Thinking, API

An explicit thinking_token_budget is off unless you set DSPARK_ENABLE_ISSUE31_GPU_HOTFIX=1 (then opt-in per request). Default stock V2 rejects the field (HTTP 400). DEFAULT_THINKING=max still needs a generous max_tokens or that budget hotfix, or thinking won't end. See Thinking-token budgets.

Serve shape (not on/off, but the knobs that change the lane)

Hotfixes and diagnostics (on by default unless you skip)

Issue #21 / #26 / #27 / #43 Python hotfixes always run at container start (they are not skipped by DSPARK_SKIP_HOTFIX). #27 + the 1024 prefill cap is why six huge cold prompts queue instead of starving decode.

Stage-C only (no-ops on Anemll 0.1.1)

These warn Unknown vLLM environment variable on the default image. They matter only after you switch DSPARK_VLLM_IMAGE to Stage-C and merge docker-compose.stage-c.override.yml:

VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK, VLLM_USE_B12X_WO_PROJECTION, VLLM_DSPARK_LOCAL_ARGMAX, VLLM_DSPARK_REPLICATE_MARKOV_W1, DSPARK_SLOT_CLAMP, and the rest of the commented Stage-C block in the example. See Optional: Stage-C / 200K-16.

---

What speed to expect

Full tables, method, and older lanes: results/RESULTS-2026-08-14.md.

On the default Anemll 1M/6 stack:

| Three Sparks (TP=3, ./start-tp3.sh, 16 slots) | Decode ≈ +4–13 % per stream and ≈ 200 tok/s aggregate at 16 streams; prefill 4–13 % slower to 64K and ≈ 22 % slower at 128K–256K (5.0 / 18.6 / 91 / 202 s TTFT at 8K / 32K / 128K / 256K vs 4.4 / 18.0 / 75 / 165 s on two nodes). See Optional: three Sparks (TP=3). |

That ~170–190 c=6 number is six streams generating, not six huge prefills. Live 2026-08-14 on this cluster: 256 × c=6 = 162 agg; 128K × c=1 still 75 tok/s / 80 s TTFT.

Live 2026-09-02, same lane, sp-indexer on, capture 48, inflight 2 (docs/CLAUDE/ab-results-2026-09-03.md): 256 × c=1 = 56 tok/s (8-trial median, 51–67), 256 × c=6 = 139 agg, 128K × c=1 TTFT 78.5 s. Decode is 15–25 % under the 14 Aug figures above and no .env.dspark knob accounts for it; treat the 14 Aug numbers as the best seen, not the norm.

315 / 205 tok/s (200K context, 16 slots) needs the Stage-C + Keys path. The ~182 1M/6 microbench was also measured with that Keys mask; the same ballpark on short prompts is already what Anemll does (1 Aug 256 × c=6 = 191 agg; 14 Aug = 162). Setting VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK=1 on Anemll 0.1.1 is a no-op (warning only). See Optional: Stage-C / 200K-16 and docs/ENVS.md.

Capture: docs/benchmarks.png.

---

Thinking and max_tokens

[!IMPORTANT] The #31 GPU budget patch is opt-in at boot (DSPARK_ENABLE_ISSUE31_GPU_HOTFIX=1) and then opt-in per request. Default is stock V2: do not send thinking_token_budget. When enabled, counters stay on the GPU; omitting the field does not inject a server-side default. Leave the flag at 0 unless a client actually sends the field (Issue #66).

max_tokens counts think + answer (reasoning + visible response + tool markup). With DEFAULT_THINKING=max, a harness cap of 256/512/800 often returns content: null / finish_reason: length because reasoning eats the whole budget — max ships a checkpoint-level directive ("do not stop reasoning until … no error remains undiscovered") and produced ~50,000 reasoning chars (~12.5k tokens) on a moderate prompt in live measurement. So "size max_tokens accordingly" means tens of thousands of tokens, not a small bump. Raise max_tokens, set thinking low / off, or — with DSPARK_ENABLE_ISSUE31_GPU_HOTFIX=1 — send an explicit thinking_token_budget (see Thinking-token budgets) so reasoning is hard-capped and the rest of max_tokens is left for the visible answer.

Client stop strings used to fire inside <think>. The recipe applies patches/hotfix-dsv4-suppress-stops-in-reasoning.py so they wait for </think>. Opt out: DSPARK_SUPPRESS_STOPS_IN_REASONING=0.

A tool call cut off by max_tokens used to report finish_reason: "tool_calls" with invalid JSON arguments and silently poison the transcript (HTTP 400 on the next turn). The recipe applies patches/hotfix-dsv4-issue55-tool-truncation.py so a truncated call reports finish_reason: "length" (not "tool_calls") and any non-JSON-parseable arguments are dropped. Clients that read length can discard the in-flight call and retry; normal model-stopped tool calls keep finish_reason: "tool_calls". Harnesses that ignore finish_reason and blindly replay args from streaming deltas can still hit a 400 - verify your client drops an in-progress tool call on finish_reason: "length".

Thinking-token budgets

max_tokens caps all new tokens (reasoning + visible answer + tool markup). thinking_token_budget caps only the reasoning portion and forces a single </think> at the boundary, leaving the rest of max_tokens available for the visible answer or tool call. A budget of 0 disables reasoning for that request. Natural </think> remains untouched.

The field is HTTP 400 on stock V2. Set DSPARK_ENABLE_ISSUE31_GPU_HOTFIX=1 and recreate the containers, then send it when a hard cap is required. Omitting it keeps DEFAULT_THINKING behavior. Keep max_tokens comfortably above the thinking budget so the answer has room.

{
  "max_tokens": 8192,
  "thinking_token_budget": 1024,
  "temperature": 0.6,
  "top_p": 0.95,
  "chat_template_kwargs": {"thinking": true, "reasoning_effort": "high"}
}

Inspect finish_reason and completion_tokens. length + null content means think ate the cap.

Enabling the budget from a client

thinking_token_budget needs the boot flag and the request field — the server injects no default when the field is omitted. To turn it on, set DSPARK_ENABLE_ISSUE31_GPU_HOTFIX=1, recreate, then send it from the client:

curl / any OpenAI-compatible client — add thinking_token_budget to the request body. 0 disables reasoning for that one call; N>0 caps reasoning at N generated tokens and leaves the rest of max_tokens for the visible answer:

curl :8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"deepseek-v4-flash-vision-exp",
  "messages":[{"role":"user","content":"Design a small rate limiter."}],
  "max_tokens":4096,
  "thinking_token_budget":1024,
  "temperature":0.6,"top_p":0.95,
  "chat_template_kwargs":{"thinking":true,"reasoning_effort":"high"}
}'

pi — the budget needs the boot flag and the pi model entry, so pi-models.dspark.example.json ships supportsThinkingTokenBudget: false to match the server default (DSPARK_ENABLE_ISSUE31_GPU_HOTFIX=0). Copy it to ~/.pi/agent/models.json; once the boot flag is 1, flip the capability on the deepseek-v4-flash-vision-exp model and pi attaches a thinking_token_budget whenever thinking is enabled:

{
  "id": "deepseek-v4-flash-vision-exp",
  "reasoning": true,
  "compat": {
    "supportsThinkingTokenBudget": true,
    "thinkingFormat": "chat-template",
    "chatTemplateKwargs": {
      "thinking":       { "$var": "thinking.enabled" },
      "reasoning_effort": { "$var": "thinking.effort", "omitWhenOff": true }
    }
  }
}

pi sizes the budget from per-level defaults (minimal 1024 / low 2048 / medium 8192 / high 16384), always leaving room for the answer; override values with the pi thinkingBudgets setting. If the capability is true while the boot flag is 0, every pi request fails with thinking_token_budget is not yet supported by the V2 model runner — so keep the capability false unless DSPARK_ENABLE_ISSUE31_GPU_HOTFIX=1.

---

How the KV cache works (why 1M + concurrency is safe)

6 ×  50k  =  300k   easy
6 × 200k  =  1.2M   fits
6 × 500k  =  3.0M   near/over pool
6 ×   1M  =  6.0M   impossible — extras queue

The boot line Maximum concurrency for 1,048,576 tokens … 2.22x only means a few simultaneous full-1M requests fit.

---

If output garbles, loops, or leaks XML

Validate direct :8888 first, then the agent harness.

1. Same image digest on both nodes (docker image inspect $DSPARK_VLLM_IMAGE). Compose must use /usr/local/bin/vllm (Anemll), not Stage-C /opt/env.
2. Full 0731 hub snapshot on head and worker, including encoding/encoding_dsv4.py.
3. Send temperature: 0 for deterministic curls. Clear harness fallback lists so another model cannot poison the transcript.

If direct vLLM is clean and the agent is not, fix the harness — do not switch to fp8 or a smaller model to hide it.

---

Optional: three Sparks (TP=3)

The default lane stays two nodes. A third DGX Spark is opt-in through its own launcher, so nothing in .env.dspark can flip the 2-node path by accident. Full details, fabric notes and the reasoning: docs/TP3.md.

Before the first boot (the launcher checks these, but does not do them):

• passwordless SSH from the head to spark3;
• the pinned DSPARK_VLLM_IMAGE already pulled on spark3 (≈ 19 GB; the launcher exits with the exact docker pull command otherwise);
• a ConnectX link head ↔ spark3 on its own /24 (e.g. 10.0.23.1 ↔ 10.0.23.3) plus a LAN interface all three nodes share for the bootstrap (default enP7s7).

Spark3 needs no local checkpoint: it mounts the head's HF cache over NFS, and the launcher creates its directory and syncs compose, env and patches/.

# .env.dspark — in addition to the 2-node settings
WORKER2_HOST=10.0.0.3
WORKER2_VLLM_HOST_IP=10.0.0.3
WORKER2_NFS_SERVER_IP=10.0.23.1        # head IP on the spark1<->spark3 link
WORKER2_NCCL_IB_HCA=rocep1s0f1         # spark3's port facing the head
WORKER2_NCCL_SOCKET_IFNAME=enp1s0f1np1
WORKER2_TP_SOCKET_IFNAME=enp1s0f1np1
WORKER2_GLOO_SOCKET_IFNAME=enp1s0f1np1
TP3_MAX_NUM_SEQS=16                    # slots on the 3-node lane only

./start-tp3.sh                         # or: ./start-tp3.sh --max-num-seqs 16
scripts/validate_tp3.sh 127.0.0.1:8888 # proves the shard, not just HTTP
./stop-deepseek-v4-flash-dspark.sh     # tears down all three ranks

The first boot compiles for several minutes (empty JIT cache on spark3, new shard shapes everywhere); wait for the health check rather than restarting. Boot log must show DSv4 TP pad: heads 64 -> 72 for TP=3 on every rank.

What you get: 16 slots, about three times the KV cache (≈ 35 GiB per rank, ≈ 5 M cached tokens), ≈ 200 tok/s aggregate at 16 streams and slightly faster decode per stream. What it costs: prefill latency, 4–13 % up to 64K and ≈ 22 % at 128K–256K, because in DeepSeek's MLA every rank holds the full latent KV and the indexer, so attention does not shrink with a third GPU, while the head pad and the three-node all-reduce add work. Single-user long-context work is better served by the 2-node lane.

---

Optional: Stage-C / 200K-16

Historical overlay image vllm-dspark-runtime:dspark-nvfp4-stage-c (./build-dspark-vllm-runtime.sh). Both nodes need that image. Merge docker-compose.stage-c.override.yml — ./start-*.sh does not add that file by itself — and uncomment the Stage-C block in .env.dspark.

Issue #27 (LONG_PREFILL_TOKEN_THRESHOLD=1024, one in-flight long prefill) still serializes huge cold prefills. Stage-C does not turn 6 × 128K into six parallel 80 s reads. Details: results/RESULTS-2026-08-14.md, docs/PATCHES.md.

Compose is Anemll-shaped (/usr/local/bin/vllm, hotfixes under /usr/local/lib/...). Treat the first Stage-C boot as an experiment.

---

Runtime flags (default compose)

• /usr/local/bin/vllm serve · TP=2 · mp · nnodes 2
• --kv-cache-dtype nvfp4_ds_mla · --block-size 256
• --max-model-len 1048576 · --max-num-seqs 6 · --max-num-batched-tokens 8192
• --long-prefill-token-threshold 1024 · --enable-chunked-prefill · --async-scheduling
• --max-cudagraph-capture-size = MAX_NUM_SEQS * (MTP_NUM_TOKENS + 1) padded to a multiple of 8 → 48 at 6×6 (plain 42 truncates to 40 and costs 12 % at c=6, measured 2026-09-02)
• --moe-backend flashinfer_b12x · --generation-config vllm
• DSpark: {"method":"dspark","num_speculative_tokens":6,"draft_sample_method":"probabilistic"}

This is the Stage C padded NVFP4 path (584-byte sparse-MLA envelope via nvfp4_ds_mla). It is not the abandoned 416-byte “true layout” experiment.

Optional GB10 hybrid plugin: ENABLE_VLLM_GB10_PATCH=1 ./start-… (--quantization modelopt_gb10_hybrid). Default off.

CI on every pull request and push to main (.github/workflows/validate.yml) is CPU-only (scripts/ci-validate.sh). Live tok/s still needs the 2× Spark pair.

Strict Responses API verification

Stateful previous_response_id continuation requires enabling the in-memory store and its exact-source bounded-store backport:

VLLM_ENABLE_RESPONSES_API_STORE=1
DSPARK_RESPONSES_STORE_MAX_ENTRIES=256

Enabled startup is fail-closed on every rank; the default remains off. See Bounded Responses API store for cap, LRU, failure, and restart semantics. The verifier treats a continuation 404 as configuration failure, not a stateless pass.

After the server is ready, run the dependency-free live verifier to check Responses text/SSE, stateful tool continuation, strict JSON schema, reasoning, invalid-field errors, appended multi-turn prefix reuse, and disconnect cleanup:

python3 scripts/verify-responses-api-live.py \
  --base-url http://127.0.0.1:8888/v1 \
  --model deepseek-v4-flash-vision-exp \
  --output results/responses-api-live.json

To exercise the terminal LRU contract, recreate every rank with DSPARK_RESPONSES_STORE_MAX_ENTRIES=2 and add --store-capacity 2 to the command. The argument must equal the server cap; values are intentionally limited to 2..16 to prevent an accidental large generation sweep.

The verifier above is the store/tool/cache and broad no-regression gate. Issue #138 has a separate mode-strict verifier for stateless full-history replay; it always sends store: false and never uses previous_response_id. On stock (default flag 0), recreate the containers and pin the reported legacy item to HTTP 400 while the complete canonical output replay remains HTTP 200:

python3 scripts/verify-issue138-responses-history-live.py \
  --base-url http://127.0.0.1:8888/v1 \
  --model deepseek-v4-flash-vision-exp \
  --expect-legacy rejected \
  --output results/issue138-stock.json

To test compatibility mode, set DSPARK_ENABLE_ISSUE138_RESPONSES_HISTORY_COMPAT=1 in .env.dspark, stop and start the pair so both containers are recreated, then require the same replay to succeed by rerunning the command above with --expect-legacy accepted and --output results/issue138-enabled.json.

The enabled run also checks assistant semantic continuity, the exact reported four-item payload, six malformed/ambiguous neighbors that must remain HTTP 400, and valid easy/canonical controls. Neither expected mode accepts the opposite outcome. Changing the flag in either direction requires recreation, not docker compose restart, because disabling it cannot undo bytes already patched in a container writable layer.

The broad verifier run intentionally creates a ~21K-token appended conversation and a forced client disconnect. Use --skip-multiturn or --skip-disconnect only when the corresponding live behavior is outside the test scope.

---

Files

---

Credits

Full list: CREDITS.md.

drowzeys ("Keys") — DSpark concurrency patch, ragged query_start_loc, nvfp4_ds_mla wiring.

@u1tra_instinct — abliterated Vision-Exp path (ABLITERATED=1), gated on drowzeys/keys-DeepSeekV4Flash-Vision-EXP-ablit.

Also: tonyd2wild, Rafael Caricio, Fraser Price, Anemll, MiaAI-Lab packaging.

Repo scripts/docs: this repo’s LICENSE. Overlay/runtime: Apache-2.0 / upstream licenses. Weights and base images have their own terms.