GLM-5.2 EXL3 TR3 3.0 bpw

This is a TP4, rank-sliced EXL3 build of zai-org/GLM-5.2, optimized for four NVIDIA Blackwell workstation GPUs. Routed MoE experts in layers 3-77 use EXL3 Trellis weights targeting 3.0 bits per weight. Accuracy-sensitive and dense components remain in BF16.

The repository payload is 332.19 GB (309.37 GiB). This format requires the custom vLLM + Sparkinfer runtime below; it is not a drop-in Transformers model. The config.json retains ModelOpt/NVFP4 compatibility metadata used by the conversion pipeline, but the routed weights are EXL3 and the required launch flag is --quantization exl3. NVFP4 in the supplied runtime refers to the KV cache, not the routed-expert weight format.

Quantization layout

The calibration manifest is included as calibration_manifest.json. It records 12,228 owner-corpus samples across general, legal, coding/agentic, and reasoning/termination axes. The model was calibrated at TP8 and packed for TP4.

Supported runtime

The tested runtime is published at verdictai/glm52-exl3-sparkinfer:

verdictai/glm52-exl3-sparkinfer:v31-gg-v20-sic3828fd-vllm0c79e41-cu132-sm120a

The supplied scripts pin the immutable registry manifest:

verdictai/glm52-exl3-sparkinfer:v31-gg-v20-sic3828fd-vllm0c79e41-cu132-sm120a@sha256:0433ae94665b769b78dd301f952d907508a3ba80bce47a1630ec20ade8812dff

It pins:

• Gilded Gnosis v20 canonical vLLM 6722c1d (dev/gilded-gnosis) + EXL3 Trellis (rebased PR #139)
• Sparkinfer v20 canonical 1a88b389 (master) + EXL3 Trellis fused arm (rebased PR #49)
• CUDA 13.2, PyTorch 2.12, CUTLASS DSL 4.6.0, FlashInfer 801d57a, NCCL 2.30.4, SM120a
• EXL3 Trellis MoE, B12X sparse MLA, DCP A2A, and MTP speculative decoding
• NVFP4 DeepSeek-MLA KV cache using calibrated outer scales

v20 changes (Gilded Gnosis v20 base)

This release rebases the EXL3 Trellis backend onto the Gilded Gnosis v20 canonical heads (vLLM 6722c1d, Sparkinfer 1a88b389). The v20 base supplies the upstreamed MTP/DCP correctness fixes — head-major cross-rank BMM (vLLM #147), MTP verifier-decode dispatch (vLLM #164), and graph-resource isolation (vLLM #149). On top of that base this image adds (vLLM PR #139 / Sparkinfer PR #49, both rebased onto v20):

• EXL3 Trellis MoE backend for rank-sliced GLM/DeepSeek routed experts.
• MTP tool-call + DSA-crash fix — the structured-output grammar advances from the authoritative step delta so tool calling engages under speculative decoding, and has_indexer is derived from index_k so MTP draft steps that skip top-k no longer trip the fused-norm-rope assertion.
• Dual-plan Trellis prefill — prefill batches route through the planned Trellis MoE.
• SM120 + B12X flattening fix — MTP next_n>2 uses the native (B, next_n) sparse-indexer path instead of the DeepGEMM next_n<=2 flatten fallback, which had corrupted MTP-3 code generation.

Validated on 4x RTX PRO 6000 Blackwell 96 GB (TP4/DCP4, MTP-3 greedy, NVFP4 KV, FULL_AND_PIECEWISE cudagraphs):

• Code generation — the previously reported ~50% syntax-error rate under MTP-3 is eliminated; generated Python/HTML is 94–100% syntactically valid across runs. The rare remaining edge is fp8-KV/DCP floating-point nondeterminism, not the prior systematic corruption.
• LAVD long-context retrieval (r10/c5, temp 0): 10/10 (6 exact, 4 near, 0 fail) — up from 8/10 (2 fails) on the prior base. The v20 canonical MTP fixes plus the flattening fix drive the fail count from 2 to 0.

Asynchronous scheduling remains disabled as a correctness guard for this DCP4/MTP path; do not enable it on this release.

Quick start

Install Docker Engine, Docker Compose v2, the NVIDIA Container Toolkit, and the Hugging Face CLI. Then download the model and start the server:

hf download brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw \
  --local-dir "$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"

cd "$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"
chmod +x server.sh
./server.sh start
./server.sh logs

The OpenAI-compatible endpoint is available locally at http://localhost:8000/v1. Here, localhost always means the machine on which the downloader starts this model; it does not refer to the model publisher's machine.

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.2-EXL3-TR3-3.0bpw",
    "messages": [{"role": "user", "content": "What is 2 + 2?"}],
    "temperature": 0,
    "max_tokens": 128
  }'

Useful controls:

./server.sh status
./server.sh logs
./server.sh restart
./server.sh stop

Direct Compose use

server.sh exports the defaults and invokes the included docker-compose.yml. To run Compose directly, set the model and cache paths:

export MODEL_DIR="$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"
export CACHE_DIR="$HOME/.cache/glm52-exl3-sparkinfer"
docker compose pull
docker compose up -d
docker compose logs --tail 100 -f glm52

The defaults can be overridden without editing the files:

The tested GPU order intentionally keeps physical GPU 3 away from TP rank 3. On another host, set CUDA_VISIBLE_DEVICES=0,1,2,3 or use the order appropriate for that machine.

The supplied Compose file binds only to loopback by default, so it does not publish the API to the LAN or internet.

Runtime validation

The exact published image completed all 81 model-shard loads, EXL3 initialization, Sparkinfer PCIe collective initialization, NVFP4 KV allocation, and full plus piecewise CUDA graph capture. The v20 base has a larger runtime footprint, so the release preset allocates 262,144 logical KV-cache tokens: 1,024 blocks x 64 tokens x DCP4, with 65,536 tokens local to each DCP rank. The configured per-request context cap is also 262,144. A GPU 0 with no other resident processes supports a larger KV pool and context (raise NUM_GPU_BLOCKS_OVERRIDE / MAX_MODEL_LEN within the KV capacity reported at startup).

Quality was validated with the release serving configuration (MTP-3, TP4/DCP4, concurrency 5):

The LAVD fail count dropping 2 → 0 reflects the v20 canonical MTP fixes plus the SM120+B12X flattening fix. Exact/near split varies run to run (fp8-KV/DCP nondeterminism); the 0-fail result is the stable signal.

The following prefill, decode, and KLD tables are indicative reference measured on the prior base; the v20 image is validated for correctness by the results above.

Cold standalone prefill results:

Sustained decode used 20-second steady-state cells after warmup, zero input context, and continuous OpenAI stream usage:

A separate 30-second C1 run measured 48.5 tok/s. No decode cell was underfilled, capacity-limited, or errored.

Five-run, 2,047-position DCP4 KLD against the same verified BF16 reference:

Full methodology, raw JSON, and copyable Rich TUI logs are in benchmarks/2026-07-22.

Source

• vLLM EXL3 integration PR
• Sparkinfer EXL3 Trellis PR
• Upstream GLM-5.2 model
• GLM-5 technical report

License

The model and this derivative are released under the MIT license. See LICENSE and the upstream model card for attribution and usage terms.

Credits and lineage

• Z.ai for GLM-5.2.
• Brandon Music for the EXL3/TR3 encoder work, owner corpus, derivative construction, runtime PR work, validation, and publication.
• turboderp for ExLlamaV3 v0.0.43 and EXL3, including the MCG trellis implementation used by this checkpoint.
• malaiwah for the MTP-78 overlay and calibration capture.
• Josh Cartu for the MTP-78 recipe and rank-sliced runtime work.
• Luke Alonso for B12X and the associated Blackwell kernel/runtime work.
• Martin Vit and yatesdr for the Infernal Invocation / RC2 image engineering credited by the upstream MTP-78 overlay.
• Special thanks to local-inference-lab for its vLLM fork, image, review, testing, and release engineering.

Canonical project attribution is recorded in the TR3 quantization provenance.