GLM5.3-Flash-E224-DGX-Spark

autotrust/GLM5.3-Flash-E224-DGX-Spark is a compact build of zai-org/GLM-5.3-Flash for desktop Blackwell systems like NVIDIA DGX Spark. It's an unofficial derivative.

It keeps 224 of the 288 routed experts in each layer by Neural Architecture Search (NAS), uses NVFP4 for the experts and still activates 18 B parameters per token. The weights take 141 GiB, which is small enough for two DGX Sparks connected by ConnectX-7 (256 GB of unified memory in total) with room left for long-context KV cache. A single 180 GB Blackwell GPU (B200/GB200) can also run it.

The original GLM-5.3-Flash MTP layer ships unmodified in mtp/ as an optional speculative-decoding draft. It gives about 1.85× single-stream decode speed at the same output quality.

The model keeps the full 154,880-token vocabulary and has the vision tower intact.

Designed for DGX Spark

DGX Spark (GB10 Grace Blackwell) has 128 GB of LPDDR5X unified memory, 273 GB/s bandwidth and native FP4 tensor cores. That profile shaped this build:

• Memory budget. Two Sparks have 256 GB together, but each node has to fit its share of the weights, the KV cache, CUDA graphs, the OS and the desktop. The unpruned NVFP4 checkpoint leaves almost no room for KV cache on each node. At 141 GiB, this model leaves roughly 40 GiB per node. That's enough for 128 K+ thinking traces at reasoning_effort=max.
• Bandwidth budget. Decode on Spark is memory-bandwidth bound. Every token still reads the same 18 B active parameters (top-8 experts + shared expert + attention), so per-token cost doesn't change. With fewer experts resident, more of the memory stays free for KV cache and batching.
• FP4 native. Routed experts use the modelopt NVFP4 format (16-element groups, e4m3 group scale, fp32 tensor scale). GB10's Blackwell tensor cores execute it natively. Attention, shared experts, embeddings and the vision tower are in BF16.
• Same quality class as the full model. On every benchmark we measured, the gap to the unpruned model is within noise, except for a few points on GPQA-Diamond (see below).

⚠️ Hardware validation status. All accuracy and throughput numbers below were measured on a single NVIDIA B200 with the same weights. The 2× DGX Spark deployment recipe below follows NVIDIA's standard two-Spark vLLM setup. Memory figures for Spark are calculated from the measured weight footprint, not measured on Spark hardware. Expect much lower absolute tokens/s on Spark than on B200, because GB10 has about 30× less memory bandwidth. Reports from Spark owners are very welcome in the Community tab.

Benchmarks

Measured on one B200 with vLLM. Sampling follows the base model's official recipe (temperature=1.0, top_p=0.95); HumanEval also uses greedy decoding. reasoning_effort is the GLM-5.3-Flash chat-template thinking budget (low / high / max).

Scoring is strict: a response that runs out of tokens before giving a final answer counts as wrong. All numbers are single runs. MoE decoding in vLLM isn't bit-deterministic, so treat ±2–3 points as noise.

Headline

At the base model's recommended thinking budget (max), GPQA-Diamond and AIME 2025 match the unpruned NVFP4 checkpoint published by RedHatAI.

Reasoning: the thinking budget matters

Token usage per question (completion tokens, thinking included):

On Spark, size --max-model-len for the effort you use. About 64 K is enough for low. Use ≥ 160 K for max; the long tail of hard problems runs past 130 K tokens.

Tool use: BFCL v4 (function calling, AST match)

BFCL was run with bfcl-eval v4 against the local OpenAI-compatible endpoint (--tool-call-parser glm47, template-default effort, --num-threads 32). multi_turn_miss_func, miss_param, long_context and the agentic web-search/memory categories weren't run.

Throughput (single B200, reference only)

reasoning_effort=low, 1,024 output tokens, short prompts, full CUDA graphs:

These numbers come from a B200 with about 8 TB/s of HBM bandwidth. A DGX Spark has 273 GB/s per node, so single-stream decode there will be much slower. Plan for interactive single-user or small-batch serving on Spark, not high-concurrency throughput.

MTP speculative decoding (optional)

The mtp/ folder holds the original GLM-5.3-Flash MTP layer, unchanged: BF16, all 288 experts, 17 GB. vLLM loads it as a separate draft model. Speculative decoding is lossless; accuracy measured with MTP on matches the runs without it, within noise.

Measured on a single B200 with reasoning_effort=low, 1,024 output tokens, served from this repository as uploaded. Text, Chinese, tool-calling and image smoke tests pass both with MTP on and off.

¹ Earlier run with identical weight files.

When to use MTP: turn it on for interactive, low-concurrency serving (1–8 streams), which is the typical DGX Spark workload. Turn it off for high-concurrency batch serving. At 32 concurrent streams, MTP lowers aggregate throughput by about 16 % and raises median TTFT from 0.8 s to 6.4 s, because the draft's extra weights shrink the KV cache: on one B200 at --gpu-memory-utilization 0.97, the KV cache drops from 1.31 M to 0.32 M tokens.

With MTP on (2 draft tokens), HumanEval greedy scored 97.0 % and GPQA-Diamond (low) scored 77.8 %, in line with the runs without MTP. Acceptance over the reasoning-heavy eval traffic was 2.36 tokens per step.

MTP is especially useful on DGX Spark. Decode there is memory-bandwidth bound, and single-user interactive use is the typical workload, which is exactly where speculative decoding helps most. The cost is 17 GB of extra weights (about 8.5 GB per node at TP=2), so leave room for it in your memory budget (see Deployment).

Where it loses vs. the unpruned model

• GPQA-Diamond at low effort: about 78 % here vs. the low-80s for larger builds. At full budget (max), the gap closes to within noise.
• Vision: MMMU val 73.6 %. We didn't measure the unpruned model in the same harness, so we can't quantify the gap.

Deployment

Requirements

• vLLM with GLM-5.3-Flash support: vLLM ≥ 0.30.0, or the official vllm/vllm-openai:glm53-flash image. Validated here on the ZJY0516/vllm@glm-release branch (vllm-project/vllm#53906) at commit 7e2d791 plus 8f8cc41 ("Make GLM-5.3 kpool metadata graph-safe"). Without that fix, full CUDA graphs can crash under concurrency.
• transformers >= 5.16.1
• Set VLLM_USE_DEEP_GEMM=0.

2× DGX Spark (target configuration)

1. Connect the two Sparks with a QSFP cable on the ConnectX-7 ports. Then follow NVIDIA's "Connect two Sparks" playbook for networking and passwordless SSH.
2. Download this repository to the same path on both nodes.
3. Launch vLLM with tensor parallelism across the two nodes. With the multiprocessing backend, run:

# on both nodes: point NCCL/Gloo at the ConnectX-7 interface (check `ibdev2netdev`)
export VLLM_USE_DEEP_GEMM=0
export NCCL_SOCKET_IFNAME=enp1s0f1np1 GLOO_SOCKET_IFNAME=enp1s0f1np1

# node 0 (head)
vllm serve autotrust/GLM5.3-Flash-E224-DGX-Spark \
  --served-model-name glm53-flash-e224 \
  --tensor-parallel-size 2 --nnodes 2 --node-rank 0 --master-addr <HEAD_CX7_IP> \
  --kv-cache-dtype fp8 --max-model-len 163840 --max-num-seqs 8 \
  --gpu-memory-utilization 0.85 \
  --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 \
  --host 0.0.0.0 --port 8000

# node 1 (worker)
vllm serve autotrust/GLM5.3-Flash-E224-DGX-Spark \
  --tensor-parallel-size 2 --nnodes 2 --node-rank 1 --master-addr <HEAD_CX7_IP> --headless \
  --kv-cache-dtype fp8 --max-model-len 163840 --max-num-seqs 8 \
  --gpu-memory-utilization 0.85

You can also use the Ray-based run_cluster.sh flow from NVIDIA's dgx-spark-playbooks vLLM guide with --tensor-parallel-size 2.

Memory per node (estimate): about 70.5 GiB of weights + about 40 GiB of KV cache and activations at --gpu-memory-utilization 0.85 (unified memory is shared with the OS). With MTP enabled, add about 8.5 GiB per node for the draft. Lower --max-model-len or raise the utilization if you don't need max effort.

Enable MTP (recommended for interactive use) by adding this to the command on both nodes:

  --speculative-config '{"method": "mtp", "model": "<local-path-to-this-repo>/mtp", "num_speculative_tokens": 2}'

If TP=2 over the interconnect is slow or unstable on your setup, try --tensor-parallel-size 1 --pipeline-parallel-size 2. Pipeline parallelism sends far less traffic over the cable per token, at the cost of single-stream latency.

Single B200 / GB200 (validated)

export VLLM_USE_DEEP_GEMM=0
vllm serve autotrust/GLM5.3-Flash-E224-DGX-Spark \
  --served-model-name glm53-flash-e224 \
  --kv-cache-dtype fp8 --max-model-len 172032 --max-num-seqs 48 \
  --gpu-memory-utilization 0.93 \
  --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45

On a 183 GB B200 this loads 141.5 GiB of weights and leaves about 14–18 GiB of KV cache: 1.1 M tokens with BF16 KV at 172 K context, or 2.5 M tokens with --kv-cache-dtype fp8. With MTP (--speculative-config '{"method": "mtp", "model": "<repo>/mtp", "num_speculative_tokens": 2}'), weights total 155.3 GiB. Use --max-model-len 66560 --gpu-memory-utilization 0.97 on a single B200.

Request format

• Thinking is always on and comes back in the reasoning / reasoning_content field.
• reasoning_effort: low, high or max (the default). Pass it as the top-level OpenAI field or via chat_template_kwargs.
• Recommended sampling: temperature=1.0, top_p=0.95. Greedy decoding can make long thinking loop on hard prompts.
• Use low for chat, Q&A, tool calls, MCQ and vision. Use max with max_tokens ≥ 131,072 for competition math and GPQA-level science.
• Tools: --tool-call-parser glm47 --enable-auto-tool-choice returns structured tool_calls.
• Images and videos use standard OpenAI multi-part content (image_url / video_url).

from openai import OpenAI
client = OpenAI(base_url="http://<head-node>:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="glm53-flash-e224",
    messages=[{"role": "user", "content": "Prove that sqrt(2) is irrational."}],
    temperature=1.0, top_p=0.95, max_tokens=32768,
    extra_body={"reasoning_effort": "high"},
)
print(r.choices[0].message.content)

Evaluation protocol

Limitations

• This is an unofficial derivative, not produced or endorsed by Z.ai or NVIDIA.
• It doesn't fit a single DGX Spark. The 128 GB unified memory is smaller than the 141 GiB of weights. You need two Sparks or a ≥180 GB GPU.
• The DGX Spark deployment isn't benchmarked yet. All measured numbers come from a B200.
• Like the base model, it can produce inaccurate, biased or unsafe content. Evaluate it for your use case before deploying.

License: MIT (inherited from the base model).