DeepSeek-V4.1-Flash on 2× DGX Spark (GB10) — our TensorFold TP2 engine

Bertholomus AI recipe: DeepSeek-V4.1-Flash served tensor-parallel over two NVIDIA DGX Spark / GB10 nodes by the model family we wrote for TensorFold (by ashhart).

Weights: Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw (MIT), an EXL3 quant by Mia-AiLab of deepseek-ai/DeepSeek-V4.1-Flash (MIT, © 2023 DeepSeek). We did not make or modify these weights; this repository holds no weights. The Engram tables are read at run time from DeepSeek's original shards 47 and 48.

Engine (v0.6.0 on): bertholomus/TensorFold, branch deepseek-v41-zig, Bertholomus AI's dsv41 family on TensorFold 1.0.2's native Zig engine. Attribution: tools/dsv41-zig/ATTRIBUTION.md. The Python engine of v0.1.0-v0.5.1 stays on branch deepseek-v41-tp2 (design, report and attribution in tools/dsv41/). Clean-room: the model math is re-implemented from DeepSeek's MIT inference code and tech report; no code from other DeepSeek-V4.1 recipes or kits was read or copied.

v0.6.1 (2026-10-09): the kernel kit in git, no engine change

The kit the Zig engine loads is now in this repository's kit/, byte for byte the kit our served lane loads (engine ce20a545, served 65f295ec), with MANIFEST, KIT-LICENSES.md and kit/tools/verify_kit.sh. The engine branch gains the builders (5e10970a: tools/dsv41-zig/build_kit.sh, build_inputs.sh); its code is unchanged. Our served lane started from this kit in a fresh folder gives the same replies as before on every gate (images 9/9 and 18/18, 51 image probes, 14 prompt lengths).

v0.6.0 (2026-10-09): the engine moves to TensorFold 1.0's native Zig engine, images included

Engine ce20a545 on branch deepseek-v41-zig, on upstream TensorFold v1.0.2. The whole dsv41 family (prompt and decode paths, DSpark drafting, Engram, the RDMA ring between the two nodes, the vision tower) now runs in TensorFold 1.0's native server, with no Python in the serving path. It is what we serve now: --context 1048576, --parallel 4, images on.

• Same replies as v0.5.1. On the same nodes, the prompts' logits of 14 prompt lengths match v0.5.1's byte for byte; the release gates (== reference 12/12, drafted == serial, concurrent == solo 12/12 burst and 12/12 staggered, images 6/6) pass; image replies match token for token (9/9 alone, 18/18 four at a time), and so do 51 image probes (formats, message layouts, refusals and their words).
• One reply fix: TF_DS_REPLAY_FLOOR=1 (on in the served configuration). A last prompt chunk of 16 rows or fewer at the bounded replay's cut (prompts of 2048k+112 to 2048k+127 tokens, or a short last chunk) read part of its attention window from the previous request's keys, so its reply could depend on what ran before it (v0.5.1 does this too). Now it reads from the cut as a larger chunk would: 14 such prompt lengths give the same reply on every pass. 2 of the 51 image probes (both in that class) change their reply.
• Images in full: the DeepSeek vision tower and aligner at full resolution, PNG decoded natively and every other format through the runtime's Pillow, the same refusals as v0.5.1.
• Memory: 5 GiB left on the head node while serving at 1M context with 4 streams.

Measured on two GB10 nodes (TP2), engine ce20a545 (code = served commit 65f295ec) at the served settings below, our own client (kit_bench.py of the deepseek-v41-tp2 branch), greedy, DSpark drafting, median of 3 unless noted. The speed rows ran before TF_DS_REPLAY_FLOOR was switched on (it changes only prompts of the class above); the gates, image checks and probes were run again live with it on: all pass.

v0.5.1's engine (6a6cf68) measured in the same session, same client and prompts, on the same nodes:

The two structured rows are inside v0.5.1's own run-to-run spread in that session (v0.5.1: 106.5-107.4 and 142.2-143.5 tok/s).

v0.5.1 (2026-10-07): fixes for reasoning loops on long agentic turns (#6), no speed change

Engine 90b68e0 on top of v0.5's 808eb4a. Replies and speed rows are unchanged on our served configuration.

• fp32 GEMMs stay fp32 in NVIDIA's PyTorch container. nvcr.io/nvidia/pytorch:26.07-py3 sets TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1, which ran the prompt chunks' fp32 router, indexer and mHC GEMMs in TF32. That gave other bits than our lane and the gates, and the loop in #6 at seed 3001. The engine now sets the switch to 0 at start (it prints a line when it was 1) and refuses to start if fp32 GEMMs still run in TF32.
• Opt-in reasoning loop guard (default off). When 3 windows of 1,024 reply tokens in a row have fewer than 2% new token 8-grams, the server closes the thinking and the reply goes on to its answer. TF_LOOP_GUARD=1 makes it the server default; a request sets it with "loop_guard": true / false. Replies that do not loop are untouched. The signal is from Capicua25x's loop detector (#9).
• Clients on long agentic turns: send the tool schemas the conversation uses. In our runs that alone avoided the loop (0 / 60 seeds).

v0.5 (2026-10-07): 1M context on the shipped configuration, faster long prompts, measured against v0.4

Decode and prefill are what we worked on this time. The same release suite on the served lane for both builds (v0.4 = engine bd0024d at --context 262144, v0.5 = 808eb4a at --context 1048576), same client (tools/dsv41/kit_bench.py), same prompts, greedy, DSpark drafting. Median of 3 unless noted.

• Context window: 262,144 → 1,048,576 tokens with --parallel 4. Needles found to 1M on this configuration; four long streams together give the same replies as each one alone.
• Decode after a 128K prompt: 89.2 → 128.7 tok/s (+44%)
• Prompt speed: 8K 1,944 → 2,125 (+9%), 32K 1,977 → 2,189 (+11%), 64K 1,897 → 2,123 (+12%), 128K 1,732 → 2,004 (+16%) tok/s
• Level: single stream set b code / prose / structured 101.1 / 62.5 / 142.4 → 100.5 / 62.5 / 140.0; 4 streams 112.5 → 112.4 (9-rep median 112.1 → 112.4); 4 streams sustained 124.5 → 125.1
• Still exact: v0.5 replies are bit-identical to v0.4.1's on every gate (== reference 12/12, drafted == serial, concurrent == solo 12/12 burst and 12/12 staggered, images 6/6). Same weights and bits, same KV cache.
• 1M qualifier (this build, this config): needles 3/3 to 1,039,836 tokens; 4 x 160K streams concurrent == solo 5/5; memory floor 6.44 / 5.44 GiB.

Where it came from, all bit-identical: RoPE tables at the context length (cos/sin only) and one shared logits buffer for the round graphs pay for the 1M window; exact pruned top-k for the long-context indexer, candidate-only reindexing and tile skipping in deep decode; prompt chunks' grouped expert launches walk a work list built on the device; kept prompts shrink to a boundary instead of being dropped. The engine is now on upstream TensorFold 0.6.6. Details in the engine commit message and tools/dsv41/REPORT.md.

v0.4.1 (2026-10-05): fix, no speed change

Engine d5d7bb3 on top of v0.4's bd0024d. The Engram read-ahead now claims its slot by index. Before, list.remove() compared id arrays of other lengths and could stop the server on long, prefix-cached agent sessions (present since v0.2). One file changed; replies and speed rows are unchanged. If you run v0.4, update to this.

Context window, v0.1.0–v0.4.1: the serve command below and our served lane run --context 262144. Earlier text here said "window up to 1,048,576": that is the engine's limit, which one earlier build tested with a single needle, not the configuration these releases were measured and shipped with.

v0.4 (2026-10-05): single stream past 100 tok/s, measured against v0.3

The same release suite, run on the served lane for both builds (v0.3 = engine bbaa6cd, v0.4 = bd0024d), same client (tools/dsv41/kit_bench.py), same prompts, greedy, DSpark drafting. Median of 3 unless noted.

• Single stream, 384 tokens (set b): code 86.0 → 101.1 tok/s (+18%) (reps 101.13 / 101.14 / 101.18), prose 53.2 → 62.5 (+18%), structured 119.4 → 142.4 (+19%)
• Single stream, 512 tokens: code 72.7 → 88.9 (+22%), prose 45.6 → 57.5 (+26%), structured 87.9 → 106.9 (+22%)
• 4 concurrent streams, 384 tokens, total tokens / wall clock: 99.4 → 112.5 tok/s (+13%); 9-rep median 112.1
• 2 concurrent streams, 384 tokens: 68.2 → 79.4 tok/s (+16%)
• 4 streams sustained (4 always in flight for 90 s): 107.5 → 124.5 tok/s (+16%), per-stream p50 37.6, first token p50 0.20 s
• Decode after a 128K prompt: 59.9 → 89.2 tok/s (+49%)
• Prompt speed: level (8K / 32K / 64K / 128K 1,944 / 1,977 / 1,897 / 1,732 tok/s)
• Start to ready: 36 s (warm restart)
• Still exact: v0.4 replies are bit-identical to v0.3's (12/12); drafted == serial; concurrent == solo 12/12 burst and 12/12 staggered; images 6/6; needles 12/12 at 8K–250K. Same weights and bits, same 262K window, same KV cache.

Where it came from, all bit-identical and each behind a switch (set it to 0 for the old path):

• Paced L2 prefetch: a side stream pulls the next kernels' weights into L2 (TF_DS_L2_PREFETCH).
• Drafter: our own tensor-core vocabulary dot per rank (bit-equal to cuBLAS on all 129,280 rows), one small argmax gather, cached bias rows for the 256 most frequent tokens; drafter pass 5.1 → ~3.3 ms.
• Verify forward: dead fp32 scratch dropped from L2, multi-row router as per-chunk sums, split mHC finish on a side stream, split q/kv rotations, register-resident attention merge, fused indexer top-k; 6-row forward 39.5 → ~36.8 ms.
• Host path: Engram reads by Linux AIO (O_DIRECT), one pinned copy per round, drafter rows absorbed behind the forward.
• Cross-node gathers: rings and flags in registered memory, one fence per staging block; ~2.9 → ~1.9 ms a round. Details in the commit message and tools/dsv41/REPORT.md.

v0.3 (2026-10-04): what changed, measured against v0.2

The same release suite, run on the served lane for both builds (v0.2 = engine 08cae28, v0.3 = bbaa6cd), same client (tools/dsv41/kit_bench.py), same prompts, greedy, DSpark drafting. Median of 3 unless noted.

• Single stream, 384 tokens (set b): code 81.9 → 86.0 tok/s (+5%), prose 49.0 → 53.2 (+9%), structured 113.1 → 119.4 (+6%)
• Single stream, 512 tokens: code 71.2 → 72.7 (+2%), prose 44.4 → 45.6 (+3%), structured 82.6 → 87.9 (+6%)
• 4 concurrent streams, 384 tokens, total tokens / wall clock: 93.0 → 99.4 tok/s (+7%); 9-rep median 93.7 → 99.5
• 2 concurrent streams, 384 tokens: 64.5 → 68.2 tok/s (+6%)
• 4 streams sustained (4 always in flight for 90 s): 104.4 → 107.5 tok/s (+3%), per-stream p50 31.6 → 32.4
• Prompt speed: unchanged (8K–128K 1,739–1,984 tok/s)
• Still exact: v0.3 replies are bit-identical to v0.2's (12/12); drafted == serial 8/8; concurrent == solo 12/12 burst and 12/12 staggered; images 6/6; needles 12/12 at 8K–250K.

Where it came from: less per-row overhead in the 1-row forward, all bit-identical. hc_post is fused into the next hc_pre, the router and indexer row matmuls are smaller, the rotations and RoPE run in neighbouring kernels' epilogues, and the indexer and compressor glue is fused. Kernels per 1-row forward: 2,387 → ~990. Every switch defaults on (TF_DS_HC_FUSED, TF_DS_ROWMM2, TF_DS_ROUND_GLUE, TF_DS_ROT_*, TF_DS_INDEXER_FUSED, TF_DS_COMP_FUSED, TF_DS_TRITON_PDL); set one to 0 to turn it off.

v0.2 (2026-10-04): what changed, measured against v0.1.0

Same nodes, same public client (tools/dsv41/kit_bench.py), same prompts and same method as v0.1.0. Greedy decode, DSpark drafting, served configuration below. Median of 3 unless noted.

• Single stream, 384 tokens (prompt set b): code 69.9 → 80.4 tok/s (+15%), prose 42.3 → 49.2 (+16%), structured 96.5 → 111.1 (+15%)
• Single stream, 512 tokens: code 60.5 → 70.8 (+17%), prose 38.0 → 44.2 (+16%), structured 74.5 → 84.6 (+14%)
• 4 concurrent streams, 384 tokens, total tokens / wall clock: 80.5 → 91–93.5 tok/s (+13–16%) (medians of four separate runs of 3–9 reps; this row varies with the server start more than the others); 256 tokens 77.5 → 85.6 (+10%)
• 4 streams sustained (new measure): 4 requests always in flight for 90 s, distinct prompts: 103 tok/s aggregate, per-stream p50 31 tok/s, first token p50 0.22 s
• Prompt speed (bounded replay): 8K 1,357 → 1,935 tok/s (+43%), 32K 1,247 → 1,953 (+57%), 64K 1,342 → 1,897 (+41%), 128K 1,262 → 1,741 (+38%)
• Start to ready (restart with the per-rank weight cache, first reply included): 65 s → 35 s (−46%)
• Quality (new): fixed, seeded subsets, greedy: MMLU-200 87.5% (thinking off) / 89.0% (on, 4,096-token cap); GSM8K-100 97.0% / 96.0%. Script: tools/dsv41/quality_eval.py (subset digest 09c2e6cc43e275af).
• Still exact: drafted == serial 8/8 at T=0; every concurrent reply bit-identical to its solo reply (12/12 burst, 12/12 staggered); solo replies identical to v0.1.0's (12/12: the new kernels are bit-identical); images 6/6; needles 12/12 at 8K–250K.

Where it came from: grouped EXL3 decode linears (884 → 350 launches a forward) and a fused grouped-expert decode path, both bit-identical; Engram reads moved off the critical path; a lighter per-round rank check; NCCL over both RoCE ports for prompt chunks; a faster weight-cache read and a shorter warm-up. Details in tools/dsv41/REPORT.md.

v0.1.0 (2026-10-04, first release)

Single stream 512 tokens code / prose / structured 60.5 / 38.0 / 74.5 tok/s (set b 69.9 / 42.3 / 96.5); 4 streams 73.2–80.5 tok/s; prefill 1,247–1,365 tok/s at 8K–128K; decode after a 128K prompt 61.8 tok/s; served window 262,144 tokens (the engine accepts up to 1,048,576; an earlier build found one needle at 1,039,833 tokens); start to ready 65 s.

Run it (v0.6.0, the Zig engine)

Build the deepseek-v41-zig branch with Zig 0.17.0 inside nvcr.io/nvidia/pytorch:26.07-py3 (CUDA nvcc, sm_121):

zig build install native -Dtarget=aarch64-linux-gnu -Doptimize=ReleaseSafe -Dnvcc=/usr/local/cuda/bin/nvcc -Dsm=121 --prefix <OUT>

That gives native/bin/tensorfold-native (the server, rank 0) and bin/tf-dsv41-lanes (rank 1, following rank 0); bin/tensorfold is a test CLI, not the server. Both ranks run in the same container image with the RoCE devices, --network host --ipc host --ulimit memlock=-1. They read a kernel kit <KIT>: the family's Triton kernels as compiled binaries (aot/, cubins/), the RoPE tables, engram.json, and vision/ (the tower's attention kernel from the container's PyTorch, and the image routing bias from DeepSeek's original weights).

The kit is in this repository's kit/ folder (plain git, with MANIFEST and KIT-LICENSES.md). Five of its files are generated, not committed: the four RoPE tables (136 MB each) and the image routing bias (43 small DeepSeek weight tensors). Make them and check every file against MANIFEST, inside the nvcr.io/nvidia/pytorch:26.07-py3 container with the deepseek-v41-tp2 engine's src/ at hand:

kit/tools/verify_kit.sh --rope <MODEL_DIR> <deepseek-v41-tp2>/src --bias <ORIGINAL_CHECKPOINT_DIR>

Or download the complete kit, generated files included, from folder kit/ of bertholomus/DeepSeek-V4.1-Flash-TensorFold-TP2-2xGB10 and run kit/tools/verify_kit.sh. Both copies hold the same bytes. To build the kit from source instead, the engine branch has tools/dsv41-zig/build_kit.sh (the kit) and tools/dsv41-zig/build_inputs.sh (each node's rank cache, token map and extension build); see their headers.

Start rank 1, then rank 0:

# rank 1
TF_DS_REPLAY_FLOOR=1 NCCL_IB_HCA=<HCA_PORT_0>,<HCA_PORT_1> NCCL_IB_GID_INDEX=5 NCCL_SOCKET_IFNAME=<IFACE> \
tf-dsv41-lanes <MODEL_DIR> <CACHE_DIR> 1 2 <HEAD_IP> 29620 <KIT> --engram <ENGRAM_DIR> --token-map <TOKEN_MAP> \
  --pool 1048576 --rdma <HCA_PORT_0>,<HCA_PORT_1> --graphs 1 --side 1 --prefetch 1 --aio 1

# rank 0
TF_DS_REPLAY_FLOOR=1 TF_TP_WORLD=2 TF_TP_MASTER=<HEAD_IP> TF_TP_PORT=29620 TF_DS_RANK_CACHE=<CACHE_DIR> \
TF_DS_KIT=<KIT> TENSORFOLD_CUDA_KERNELS=<KIT>/aot TF_DS_ENGRAM=<ENGRAM_DIR> TF_DS_TOKEN_MAP=<TOKEN_MAP> \
TF_RDMA_DEVICES=<HCA_PORT_0>,<HCA_PORT_1> TF_DS_GRAPHS=1 TF_DS_HC_SIDE=1 TF_DS_L2_PREFETCH=1 TF_DS_ENGRAM_AIO=1 \
NCCL_IB_HCA=<HCA_PORT_0>,<HCA_PORT_1> NCCL_IB_GID_INDEX=5 NCCL_SOCKET_IFNAME=<IFACE> \
tensorfold-native serve <MODEL_DIR> --host 127.0.0.1 --port 18891 --name DeepSeek-V4.1-Flash-TF --api-key-file <KEY_FILE> \
  --context 1048576 --parallel 4 --thinking --max-tokens 32768 --temperature 1.0 --top-p 0.95 --top-k 20 --backend cuda

• TF_DS_REPLAY_FLOOR=1 goes to both ranks.
• Rank 1 must match rank 0's settings: TF_DS_STREAMS=N on rank 0 needs --streams N on rank 1, and rank 1's --pool equals rank 0's --context. With different values the two ranks wait on each other (#25).
• Images in formats other than PNG need python3 with Pillow in the container (the NVIDIA image has it).
• <CACHE_DIR> keeps each rank's weights in one file (written on the first start); <TOKEN_MAP> and the cache are the ones the deepseek-v41-tp2 engine writes.
• TF_RDMA_GID_INDEX=<index> picks the RoCEv2 GID index the native TP2 ring opens (default 5, so existing setups are unchanged; single-rail ASUS GX10 pairs want 3). Find the index with ibv_devinfo -v or show_gids, and set it on both ranks. Both ranks must run the same engine build: the ring's peer description grew in the same change (128 to 144 bytes), and a mixed pair cannot talk. At connect the ring logs gid_index= and the negotiated path_mtu= per rail.

Hosts and addresses are placeholders; substitute your own.

Run it (v0.1.0-v0.5.1, the Python engine)

Two GB10 nodes with a direct RoCE link. On each node, a throwaway nvcr.io/nvidia/pytorch:26.07-py3 container with the deepseek-v41-tp2 branch installed (pip install -e). Start rank 0 (head) first — it owns the TCPStore master (comm.py: is_master = rank == 0) — then wait for its store port to LISTEN (default 29551) before starting rank 1 (worker). A fixed sleep races on a cold start: rank 0 dies with DistStoreError: Timed out … 1/2 clients joined and rank 1 with NCCL error 6 at nccl.barrier():

TF_DS_REPLAY=1 TF_DS_PREFILL_CHUNK=2048 TF_DS_RANK_CACHE=<CACHE_DIR> TF_DS_RANK_CACHE_READERS=32 \
TF_DS_WARM_LENGTHS=1,17,33,131,514,1024,2113 \
NCCL_IB_HCA=<HCA_PORT_0>,<HCA_PORT_1> NCCL_IB_GID_INDEX=5 NCCL_SOCKET_IFNAME=<IFACE> \
tensorfold serve <MODEL_DIR> --tp 2 --rank R --master <HEAD_IP> --host 127.0.0.1 --port 18891 \
  --context 1048576 --vision --parallel 4 --mtp-drafts 5

• <MODEL_DIR>: the Mia-AiLab EXL3 checkpoint. Engram tables: TF_DS_ENGRAM=<ENGRAM_DIR> (DeepSeek's original shards 47/48), or a folder next to <MODEL_DIR> whose name contains "Engram".
• NCCL_IB_HCA lists both RoCE ports when both are cabled (prompt-chunk gathers 153 → 106 ms); one port works too.
• TF_DS_RANK_CACHE keeps each rank's weights in one file (~106 GB a rank, written on the first start).
• --context 1048576 is the measured, served window from v0.5 (v0.1.0–v0.4.1 ran 262144).
• TF_RDMA_DEVICES=<HCA_PORT_0>,<HCA_PORT_1> names the RoCE ports of the link between the two nodes when a node has more active ports than that link (the RDMA gather needs the same device count on every rank). TF_API_KEY_FILE=<file> makes every route but /health require a key.
• --temperature sets the server-side default a client inherits when it omits the field. Greedy (0) makes long agentic turns loop in reasoning_content and return empty content (finish_reason: "length"); omit it to keep the engine default (1.0), and pin 0 only for the deterministic MMLU/GSM8K runs.
• Benchmark: python3 tools/dsv41/kit_bench.py --base http://127.0.0.1:18891 --model <name> decode|concurrent|sustained|prefill|depth
• Quality: python3 tools/dsv41/quality_eval.py --base http://127.0.0.1:18891 --model <name> --mmlu <MMLU_TEST_PARQUET> \ --gsm8k <GSM8K_TEST_JSONL> --max-tokens-on 4096 (MMLU "all" test from cais/mmlu, GSM8K test from openai/grade-school-math)

Hosts and addresses are placeholders; substitute your own.

Credits

• DeepSeek — DeepSeek-V4.1-Flash, its inference code and tech report (MIT, © 2023 DeepSeek).
• Mia-AiLab — the EXL3 2.9 bpw quant this recipe serves (MIT).
• ashhart — TensorFold (Apache-2.0), the engine this family plugs into.
• turboderp — EXL3 / exllamav3 (MIT), the weight format.
• Capicua25x — the #6 report and repro, and the 8-gram novelty loop signal (#9).
• Bertholomus AI (Albert Lee, bertholomus) — the deepseek_v41 TP2 family and its Zig port dsv41, their kernels, the deployment and the measurements.

Not affiliated with or endorsed by DeepSeek, NVIDIA, the TensorFold authors, Mia-AiLab or MiaAI-Lab.

License

Recipe documentation: Apache-2.0 (LICENSE). The weights stay under their own MIT license.