Use the TensorFold recipe

[!IMPORTANT]

There is an improved version of this recipe that runs faster on TensorFold. Everyone should use that.

https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold

GLM-5.3 Flash EXL3 on DGX Spark

Serve GLM-5.3 Flash on 2, 3 or 4 NVIDIA DGX Sparks with one command.
_{by Mia's AI Lab and the community}

Chris Scott (chriswritescode-dev) authored the root commit dc6b4fdd.

An OpenAI-compatible vLLM server for zai-org/GLM-5.3-Flash, using 4-bpw EXL3 weights, fp8 KV cache and DFlash2 speculative decoding, built natively for the GB10 (sm_121a). The launcher pulls the image, downloads the weights, shares them with the other Sparks and starts the cluster.

At a glance

Quick start

On the head Spark, with the second Spark reachable over SSH and the CX7 cable connected:

git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks.git
cd GLM-5.3-Flash-EXL3-2x-DGX-Sparks
cp .env.example .env     # set HEAD_IP, WORKER_IP and the CX7 interface names
./start.sh               # pull, download, share weights, start, warm up

The first start downloads ~164 GiB and takes a while. When it prints GLM-5.3-Flash EXL3 is UP, send a request:

curl -s http://127.0.0.1:8888/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model": "GLM-5.3-Flash-EXL3", "messages": [{"role": "user", "content": "Hello!"}]}'

[!NOTE] After a git pull that changes the image recipe, the next start rebuilds the image once. That is expected.

Network setup, NFS weight sharing, air-gapped installs and other kits: Quick start reference · Running on a different kit.

Performance

Latest TP2 measurements on 2× GB10, 2026-09-28: DFlash2, temperature 0, thinking off, mean of fresh boots. Decode excludes time to first token.

[!IMPORTANT] Default is what every install gets. Speed boost is opt-in: one .env line and a one-time ~25 min build on the head. See TP2 profiles.

xychart-beta
    title "TP2 decode (tokens/s): default vs speed boost"
    x-axis ["Prose", "Structured", "Code", "Code 32k", "Short code x1", "Short code x2"]
    y-axis "tokens/s" 0 --> 100
    bar [37.5, 92.3, 55.5, 60.3, 41.3, 56.9]
    line [36.1, 83.0, 54.6, 54.6, 36.0, 53.6]

_{Bars: speed boost. Line: default.}

Recent gains

• Default, 2026-09-26: decode 3.9–5.2% faster and 16k/64k time to first token 11.5–12.6% lower than the previous default, with the same weights and a numerics check that found no quality loss. (finding)
• 4× Spark, 2026-09-27: faster prefill is now on by default: 1,772 / 2,422 / 2,733 tokens/s cold prefill at 8k / 32k / 100k, up from 1,316 / 1,761 / 1,940 (contributor measurement).
• Abliteration + speed boost, 2026-09-30: decode 34.97 / 82.84 / 51.28 tokens/s (prose / structured / code) against 34.87 / 80.68 / 46.54 for plain abliteration: up to +10%, one boot per arm.

[!NOTE] Speed-boost figures come from 2–3 boots per arm with unlocked clocks; treat them as descriptive, not confidence intervals. Methods, raw receipts and older tables: Performance history.

TP2 profiles

Pick one per start. All run on the same public base weights.

[!TIP] Pass ABLIT=1 on the command line, not in .env: the launcher resets it unless you ask for it at start.

Details: Speed boost · Abliteration · Both together

Optional tuning

Off by default. Each has a measured benefit and a tradeoff; try one at a time.

All options and their tradeoffs: full reference.

Topologies

Stop one topology before starting another; they share port 8888.

Configuration

.env holds everything; the shipped defaults are tuned for 2× Spark. A variable set on the command line wins over .env (MAX_NUM_SEQS=2 ./start.sh restart).

Using the API. Thinking is on by default. For long structured output, prefer "chat_template_kwargs": {"reasoning_effort": "low"} over turning thinking off; thinking off can corrupt long numeric tables. Responses report cached prompt tokens in usage.

Every knob with its measured tradeoff: .env reference · Client request defaults · KV cache · Prefix caching.

[!WARNING] Do not use --moe-backend marlin, NVFP4 or bf16 KV cache, or TRITON_ATTN for the draft. They fail or silently degrade on this image. Full list: Do not.

For agents and contributors

• Start here, then go deep: docs/REFERENCE.md is the complete reference, including what runs, why the overlay exists and the image build.
• What changed and when: CHANGELOG.md.
• Design notes and measurements: docs/.
• Tests: CPU tests live in tests/ and run with python3 -m pytest tests/ or directly. tests/test_image_layer_budget.py keeps the image loadable on workers; fold new Dockerfile steps into existing layers.
• Pull requests: describe what you measured and on which kit. Defaults change only with receipts.

License and credits

Recipe code and docs: AGPL-3.0 (contributions before 2026-09-07: MIT). Weights keep their own licenses: the EXL3 checkpoint is Apache-2.0 and DFlash2 is CC BY-NC-ND 4.0.

Chris Scott (chriswritescode-dev) authored the root commit dc6b4fdd.

tonyd2wild (profile) authored overlay/patch_glm5_drafter_group.py, which adds the DFlash2 sliding-window KV group to the GLM-5-Next layout, in commit 64c92e98 of GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark. This repo's first copy of the file is his, byte for byte; later changes are ours.

Built on the work of turboderp (ExLlamaV3), Z.ai (base model), IncoAI (DFlash2), Alexbob0 and vcruz305 (dense EXL3 loader), drowzeys and bullerwins (abliteration), FlyCockpit (TP3 shapes), malaiwah (KLD panel), and every contributor who sent a fix or a measurement. Full credits.