DeepSeek V4.1 Flash 1017 on dual DGX Spark: TensorFold and vLLM recipes, matching Engrams, and C1–C8 thinking off/75 prefill and decode metrics.
Python
6
10 commits
updated Oct 18, 2026
A pinned deployment recipe for LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw across two DGX Spark / GB10 systems. It provides separate TensorFold and vLLM profiles using the same checkpoint and official 1017 Engram tables, plus a reproducible C1–C8 prefill and decode benchmark with thinking disabled and thinking effort 75.
The tested TensorFold path is its Python 0.6.0 engine with Jay Leaton's DeepSeek family patches and the small 1017 adaptations in this repository. The upstream Zig engine and 2.8bpw pack are separate deployment paths. Results here identify the engine, weights, thinking mode and measurement conditions explicitly.
TensorFold is faster for single-request decode. vLLM is faster for effective prefill in these quick runs.
Both engines completed 144 requests each across C1–C8, thinking disabled and reasoning effort 75, with separate prefill and decode runs. Prompt hashes and input/output token counts match across engines. Full comparison and raw records.
Decode is single-request C1 output tok/s, measured after the first token and excluding initial prefill. Prefill is the best measured effective input tok/s from C1–C8; its concurrency is shown beside the rate. Prefill includes scheduling and first-token delivery.
| Engine | Thinking | Raw decode (tok/s, C1) | Best prefill (input tok/s) |
|---|---|---|---|
| TensorFold | off | 58.1 | 451.2 (C2) |
| TensorFold | 75 | 77.5 | 444.0 (C7) |
| vLLM | off | 36.9 | 850.8 (C6) |
| vLLM | 75 | 41.4 | 840.5 (C6) |
TensorFold leads single-request decode in both thinking modes. vLLM leads effective prefill. These are the best observed prefill rates from one short pass, not sustained-throughput estimates. Full C1–C8 measurements and raw records retain the complete comparison.
These are short, single-pass endpoint measurements, not a quality evaluation or an estimate of sustained long-response performance. Both engines use the same checkpoint. See the method for decode-window rates, per-stream timing and limitations. Decode tests cap output at 64 tokens; separate prefill tests cap it at one token.
Keep Engram lookup local to each rank. The checkpoint and Engram source revisions are pinned in config/manifest.json. The two large Engram tables are from DeepSeek-V4.1-Flash-1017, matched to this checkpoint.
bash scripts/fetch-sources.sh
docker build -t deepseek41-tensorfold:1017 .
docker save deepseek41-tensorfold:1017 | ssh spark-b docker load
The Dockerfile uses NVIDIA PyTorch 26.07, preserves its torch and Triton versions, and installs the tested TensorFold source export. The source script applies upstream patches before the local 1017 patch. It leaves the original source checkouts available for review.
Set TF_DSV41_BIAS_VL=/model on both ranks: this 1017 checkpoint contains the image-router biases inline. The preflight reads all 43 bias tensors and checks that the native attachment path loads the 40 main-layer routers correctly.
The local adaptations cover 1017's inline image-router biases, 6-bit dense MTP layers in the vLLM overlay, tool-only turn handling, the actual TensorFold shared-pool memory budget, revision-specific packed paths and repeatable metadata publication to the NFS share.
python3 -m venv .venv
. .venv/bin/activate
pip install huggingface_hub==2.2.0 hf-xet==1.7.0
python scripts/download.py --root /srv/models/deepseek-v41-1017
The downloader verifies Hub LFS SHA-256 checksums, records a manifest and supplies the reference-tested chat template. The checkpoint does not supply a Hugging Face chat-template file; the TensorFold HTTP app needs one during initialization, even though its native encoder renders actual DeepSeek prompts.
Configure build/mia-runtime/.env using the deployment example and the tuning values in config/vllm-knobs.env.example. Set the new bundle paths and EXPECTED_SHARDS=43, then run:
bash build/mia-runtime/start.sh prepare
bash build/mia-runtime/start.sh pack
prepare publishes the source files and writes launch scripts without starting GPU servers. pack creates engram-l1-r0of2.bin and engram-l14-r0of2.bin on the head, and the rank 1 equivalents on the worker. Use unique directories for this model revision. Each file uses the same hash-head partitioning expected by both runtimes.
Check the packs with the upstream TensorFold checker; it compares headers, sizes and sampled records with the verified source:
python build/jayleaton/scripts/pack_engram.py \
--src /srv/models/deepseek-v41-1017/engram-src-1017 \
--config /srv/models/deepseek-v41-1017/checkpoint/config.json \
--check /srv/deepseek-v41/engram-packed-1017 --rank 0
Run the rank 1 check on the worker with its source NFS volume mounted read-only. Both packs were checked with both runtimes' readers during deployment.
Copy config/cluster.env.example to config/cluster.env and set the real host, fabric and storage values. The worker model setting accepts a Docker NFS volume name. Algorithm settings are in config/tensorfold-knobs.env.example.
The deployment guide includes CPU-only extension precompilation and copying the compiled cache to the second identical GB10 system. Precompile before loading the model so compilation does not compete with the resident weights for memory.
python scripts/cluster.py check
python scripts/cluster.py args
python scripts/cluster.py start
python scripts/cluster.py health
The launcher uses the same container name on both nodes, starts rank 1 before rank 0 and affects only those containers. It refuses startup while another GPU workload is running. Stop the existing inference service explicitly before switching engines. Initial weight loading and graph warmup take several minutes.
To stop this profile:
python scripts/cluster.py stop
For vLLM, the patched Mia kit provides bash build/mia-runtime/start.sh start. Apply the pinned image digest and tuning file first. Dashboard integration describes the two-selector layout used on port 11001.
Run one engine at a time with other API clients paused:
python scripts/benchmark.py \
--model deepseek-v4.1-flash-1017-tensorfold --label tensorfold \
--thinking off --input-lines 27 --output-tokens 64 --repeats 1 --skip-warmup --out results/raw-local/tensorfold-off
python scripts/benchmark.py \
--model deepseek-v4.1-flash-1017-tensorfold --label tensorfold \
--thinking 75 --input-lines 27 --output-tokens 64 --repeats 1 --skip-warmup --out results/raw-local/tensorfold-thinking75
Use the vLLM model ID deepseek-v4.1-flash-1017-exl3 with the same arguments after switching. For prefill-only tests, add --output-tokens 1. The complete matrix, token lengths, counting conventions and limitations are documented in the method.
This repository distributes the integration recipe, source patches and benchmark artifacts. Model weights and private machine configuration are downloaded or supplied separately. Upstream components retain their existing licenses, included in LICENSES/; no blanket license is added for original integration material.
DeepSeek V4.1 Flash 1017 on dual DGX Spark: TensorFold and vLLM recipes, matching Engrams, and C1–C8 thinking off/75 prefill and decode metrics.
Python
6
10 commits
updated Oct 18, 2026
A pinned deployment recipe for LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw across two DGX Spark / GB10 systems. It provides separate TensorFold and vLLM profiles using the same checkpoint and official 1017 Engram tables, plus a reproducible C1–C8 prefill and decode benchmark with thinking disabled and thinking effort 75.
The tested TensorFold path is its Python 0.6.0 engine with Jay Leaton's DeepSeek family patches and the small 1017 adaptations in this repository. The upstream Zig engine and 2.8bpw pack are separate deployment paths. Results here identify the engine, weights, thinking mode and measurement conditions explicitly.
TensorFold is faster for single-request decode. vLLM is faster for effective prefill in these quick runs.
Both engines completed 144 requests each across C1–C8, thinking disabled and reasoning effort 75, with separate prefill and decode runs. Prompt hashes and input/output token counts match across engines. Full comparison and raw records.
Decode is single-request C1 output tok/s, measured after the first token and excluding initial prefill. Prefill is the best measured effective input tok/s from C1–C8; its concurrency is shown beside the rate. Prefill includes scheduling and first-token delivery.
| Engine | Thinking | Raw decode (tok/s, C1) | Best prefill (input tok/s) |
|---|---|---|---|
| TensorFold | off | 58.1 | 451.2 (C2) |
| TensorFold | 75 | 77.5 | 444.0 (C7) |
| vLLM | off | 36.9 | 850.8 (C6) |
| vLLM | 75 | 41.4 | 840.5 (C6) |
TensorFold leads single-request decode in both thinking modes. vLLM leads effective prefill. These are the best observed prefill rates from one short pass, not sustained-throughput estimates. Full C1–C8 measurements and raw records retain the complete comparison.
These are short, single-pass endpoint measurements, not a quality evaluation or an estimate of sustained long-response performance. Both engines use the same checkpoint. See the method for decode-window rates, per-stream timing and limitations. Decode tests cap output at 64 tokens; separate prefill tests cap it at one token.
Keep Engram lookup local to each rank. The checkpoint and Engram source revisions are pinned in config/manifest.json. The two large Engram tables are from DeepSeek-V4.1-Flash-1017, matched to this checkpoint.
bash scripts/fetch-sources.sh
docker build -t deepseek41-tensorfold:1017 .
docker save deepseek41-tensorfold:1017 | ssh spark-b docker load
The Dockerfile uses NVIDIA PyTorch 26.07, preserves its torch and Triton versions, and installs the tested TensorFold source export. The source script applies upstream patches before the local 1017 patch. It leaves the original source checkouts available for review.
Set TF_DSV41_BIAS_VL=/model on both ranks: this 1017 checkpoint contains the image-router biases inline. The preflight reads all 43 bias tensors and checks that the native attachment path loads the 40 main-layer routers correctly.
The local adaptations cover 1017's inline image-router biases, 6-bit dense MTP layers in the vLLM overlay, tool-only turn handling, the actual TensorFold shared-pool memory budget, revision-specific packed paths and repeatable metadata publication to the NFS share.
python3 -m venv .venv
. .venv/bin/activate
pip install huggingface_hub==2.2.0 hf-xet==1.7.0
python scripts/download.py --root /srv/models/deepseek-v41-1017
The downloader verifies Hub LFS SHA-256 checksums, records a manifest and supplies the reference-tested chat template. The checkpoint does not supply a Hugging Face chat-template file; the TensorFold HTTP app needs one during initialization, even though its native encoder renders actual DeepSeek prompts.
Configure build/mia-runtime/.env using the deployment example and the tuning values in config/vllm-knobs.env.example. Set the new bundle paths and EXPECTED_SHARDS=43, then run:
bash build/mia-runtime/start.sh prepare
bash build/mia-runtime/start.sh pack
prepare publishes the source files and writes launch scripts without starting GPU servers. pack creates engram-l1-r0of2.bin and engram-l14-r0of2.bin on the head, and the rank 1 equivalents on the worker. Use unique directories for this model revision. Each file uses the same hash-head partitioning expected by both runtimes.
Check the packs with the upstream TensorFold checker; it compares headers, sizes and sampled records with the verified source:
python build/jayleaton/scripts/pack_engram.py \
--src /srv/models/deepseek-v41-1017/engram-src-1017 \
--config /srv/models/deepseek-v41-1017/checkpoint/config.json \
--check /srv/deepseek-v41/engram-packed-1017 --rank 0
Run the rank 1 check on the worker with its source NFS volume mounted read-only. Both packs were checked with both runtimes' readers during deployment.
Copy config/cluster.env.example to config/cluster.env and set the real host, fabric and storage values. The worker model setting accepts a Docker NFS volume name. Algorithm settings are in config/tensorfold-knobs.env.example.
The deployment guide includes CPU-only extension precompilation and copying the compiled cache to the second identical GB10 system. Precompile before loading the model so compilation does not compete with the resident weights for memory.
python scripts/cluster.py check
python scripts/cluster.py args
python scripts/cluster.py start
python scripts/cluster.py health
The launcher uses the same container name on both nodes, starts rank 1 before rank 0 and affects only those containers. It refuses startup while another GPU workload is running. Stop the existing inference service explicitly before switching engines. Initial weight loading and graph warmup take several minutes.
To stop this profile:
python scripts/cluster.py stop
For vLLM, the patched Mia kit provides bash build/mia-runtime/start.sh start. Apply the pinned image digest and tuning file first. Dashboard integration describes the two-selector layout used on port 11001.
Run one engine at a time with other API clients paused:
python scripts/benchmark.py \
--model deepseek-v4.1-flash-1017-tensorfold --label tensorfold \
--thinking off --input-lines 27 --output-tokens 64 --repeats 1 --skip-warmup --out results/raw-local/tensorfold-off
python scripts/benchmark.py \
--model deepseek-v4.1-flash-1017-tensorfold --label tensorfold \
--thinking 75 --input-lines 27 --output-tokens 64 --repeats 1 --skip-warmup --out results/raw-local/tensorfold-thinking75
Use the vLLM model ID deepseek-v4.1-flash-1017-exl3 with the same arguments after switching. For prefill-only tests, add --output-tokens 1. The complete matrix, token lengths, counting conventions and limitations are documented in the method.
This repository distributes the integration recipe, source patches and benchmark artifacts. Model weights and private machine configuration are downloaded or supplied separately. Upstream components retain their existing licenses, included in LICENSES/; no blanket license is added for original integration material.