Qwen3.6-35B-A3B NVFP4, NInfer v3 artifact
1
3 commits
2 linked in READMEs
updated Oct 6, 2026
A NInfer v3 artifact of
Qwen3.6-35B-A3B whose routed and shared experts are
the NVFP4 weights of RedHatAI/Qwen3.6-35B-A3B-NVFP4,
copied code for code, for NInfer-all, the master branch of
iamwavecut/ninfer-all. NVFP4 expert banks prefill
through W4A4 on Blackwell's FP4 tensor cores, so this file needs an sm_120a GPU (RTX 50 series,
RTX PRO 6000 Blackwell) and a 120a build of master from commit
a00d1636 on (earlier commits run the
banks only with -DNINFER_SM120_NATIVE=ON); sm_80, sm_86 and sm_89 builds refuse them.
| component | representation |
|---|---|
| routed experts (256 per layer) and shared expert, 40 layers | NVFP4 from RedHatAI's checkpoint: FP4 codes with an FP8 scale per 16 weights and one FP32 divisor per stacked matrix, copied without requantization |
| attention (10 layers) and GDN (30 layers) projections | Q8, groups of 32, from the BF16 base |
| router, shared-expert score, GDN A/B controls, norms, convolution | BF16; A_log and dt_bias FP32 |
| token table, output head | Q8 (groups of 32), Q6 (groups of 64) |
| MTP head | Q8, its experts included, from the base |
| proposal head | the output head's rows for the 131,072 most frequent tokens of NInfer's token ranking, Q4 |
| Vision tower | groupwise Q4/Q5, Q8 merger, Q6 patch embedding |
| chat template | NInfer's pinned qwen3_6.jinja |
One .ninfer file of 21,895,523,588 bytes (20.39 GiB; 19.13 GiB of text weights are resident
without Vision and drafters), next to its conversion report, SHA256SUMS and NOTICE.
Conversion command, from the repository's tree, with the BF16 base at revision 995ad96e and
RedHatAI's checkpoint at e94a930c:
python3 -m tools.convert \
--model Qwen3.6-35B-A3B \
--recipe qwen3_6_35b_a3b_nvfp4 \
--source quantized=Qwen3.6-35B-A3B-NVFP4 \
--components text,vision,mtp \
--resource chat_template.jinja=tools/chat_templates/qwen3_6.jinja \
--proposal \
--name qwen3.6-35b-a3b \
--out Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer
--device cpu converts without a GPU, in about four minutes.
The whole 262,144-token window with MTP and Vision, from the
Docker image, whose 120a build starts on these
GPUs (an NVIDIA driver of the CUDA 13 branch, 580 or newer, and the NVIDIA Container Toolkit):
hf download WaveCut/Qwen3.6-35B-A3B-NVFP4-NInfer-v3 Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer --local-dir models
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
-v "$PWD/models:/models" -v ninfer-cache:/cache \
ghcr.io/iamwavecut/ninfer-all serve /models/Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer \
--model-id qwen3.6-35b-a3b --max-context 262144 --kv-capacity 262144 --kv-dtype rk8v4 \
--gdn-state-fp16 --spec mtp --draft-tokens 3 --lm-head-draft --vision
The server answers OpenAI and Anthropic requests on http://localhost:8080/v1. A build of master
with CMAKE_CUDA_ARCHITECTURES=120a and CUDA 13.1 or newer
(its README) takes the same flags:
ninfer-serve models/Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer --model-id qwen3.6-35b-a3b \
--max-context 262144 --kv-capacity 262144 --kv-dtype rk8v4 --gdn-state-fp16 \
--spec mtp --draft-tokens 3 --lm-head-draft --vision
On an RTX 5090 this starts in 17 seconds and leaves 7.4 GiB of the card free.
Perplexity over the quick subset of NInfer's evaluation corpus (ninfer-perplexity --quick,
261,167 scored tokens, INT8 KV), both artifacts on the same RTX PRO 6000:
| model | perplexity |
|---|---|
| this artifact | 4.410 |
| official Qwen3.6-35B-A3B NInfer artifact (groupwise Q4/Q5 experts) | 4.364 |
An RTX 5090 scores this artifact at the same 4.410.
September 2026, ninfer_bench on NInfer's benchmark corpus, BF16 KV, greedy, three repetitions,
the default 120a build with each card's built-in device profile:
| RTX 5090 (550 W) | RTX PRO 6000 (600 W) | official artifact, RTX PRO 6000 | |
|---|---|---|---|
| prompt, 512 tokens | 21,622 tok/s | 21,705 tok/s | 18,040 tok/s |
| prompt, 4,096 tokens | 27,412 tok/s | 30,938 tok/s | 22,643 tok/s |
| decode, 128 tokens | 407 tok/s | 399 tok/s | 398 tok/s |
| decode with MTP (3 drafts, proposal head) | 541 tok/s |
The MTP row drafts over the benchmark corpus's prose (42% of drafts accepted); a short chat answer decodes at 742 tok/s on the RTX 5090. Decode reads about the same bytes as the official artifact, so it runs at the same speed; prompts gain from the FP4 tensor cores.
Apache 2.0, as both sources; see NOTICE.
Qwen3.6-35B-A3B NVFP4, NInfer v3 artifact
1
3 commits
2 linked in READMEs
updated Oct 6, 2026
A NInfer v3 artifact of
Qwen3.6-35B-A3B whose routed and shared experts are
the NVFP4 weights of RedHatAI/Qwen3.6-35B-A3B-NVFP4,
copied code for code, for NInfer-all, the master branch of
iamwavecut/ninfer-all. NVFP4 expert banks prefill
through W4A4 on Blackwell's FP4 tensor cores, so this file needs an sm_120a GPU (RTX 50 series,
RTX PRO 6000 Blackwell) and a 120a build of master from commit
a00d1636 on (earlier commits run the
banks only with -DNINFER_SM120_NATIVE=ON); sm_80, sm_86 and sm_89 builds refuse them.
| component | representation |
|---|---|
| routed experts (256 per layer) and shared expert, 40 layers | NVFP4 from RedHatAI's checkpoint: FP4 codes with an FP8 scale per 16 weights and one FP32 divisor per stacked matrix, copied without requantization |
| attention (10 layers) and GDN (30 layers) projections | Q8, groups of 32, from the BF16 base |
| router, shared-expert score, GDN A/B controls, norms, convolution | BF16; A_log and dt_bias FP32 |
| token table, output head | Q8 (groups of 32), Q6 (groups of 64) |
| MTP head | Q8, its experts included, from the base |
| proposal head | the output head's rows for the 131,072 most frequent tokens of NInfer's token ranking, Q4 |
| Vision tower | groupwise Q4/Q5, Q8 merger, Q6 patch embedding |
| chat template | NInfer's pinned qwen3_6.jinja |
One .ninfer file of 21,895,523,588 bytes (20.39 GiB; 19.13 GiB of text weights are resident
without Vision and drafters), next to its conversion report, SHA256SUMS and NOTICE.
Conversion command, from the repository's tree, with the BF16 base at revision 995ad96e and
RedHatAI's checkpoint at e94a930c:
python3 -m tools.convert \
--model Qwen3.6-35B-A3B \
--recipe qwen3_6_35b_a3b_nvfp4 \
--source quantized=Qwen3.6-35B-A3B-NVFP4 \
--components text,vision,mtp \
--resource chat_template.jinja=tools/chat_templates/qwen3_6.jinja \
--proposal \
--name qwen3.6-35b-a3b \
--out Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer
--device cpu converts without a GPU, in about four minutes.
The whole 262,144-token window with MTP and Vision, from the
Docker image, whose 120a build starts on these
GPUs (an NVIDIA driver of the CUDA 13 branch, 580 or newer, and the NVIDIA Container Toolkit):
hf download WaveCut/Qwen3.6-35B-A3B-NVFP4-NInfer-v3 Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer --local-dir models
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
-v "$PWD/models:/models" -v ninfer-cache:/cache \
ghcr.io/iamwavecut/ninfer-all serve /models/Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer \
--model-id qwen3.6-35b-a3b --max-context 262144 --kv-capacity 262144 --kv-dtype rk8v4 \
--gdn-state-fp16 --spec mtp --draft-tokens 3 --lm-head-draft --vision
The server answers OpenAI and Anthropic requests on http://localhost:8080/v1. A build of master
with CMAKE_CUDA_ARCHITECTURES=120a and CUDA 13.1 or newer
(its README) takes the same flags:
ninfer-serve models/Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer --model-id qwen3.6-35b-a3b \
--max-context 262144 --kv-capacity 262144 --kv-dtype rk8v4 --gdn-state-fp16 \
--spec mtp --draft-tokens 3 --lm-head-draft --vision
On an RTX 5090 this starts in 17 seconds and leaves 7.4 GiB of the card free.
Perplexity over the quick subset of NInfer's evaluation corpus (ninfer-perplexity --quick,
261,167 scored tokens, INT8 KV), both artifacts on the same RTX PRO 6000:
| model | perplexity |
|---|---|
| this artifact | 4.410 |
| official Qwen3.6-35B-A3B NInfer artifact (groupwise Q4/Q5 experts) | 4.364 |
An RTX 5090 scores this artifact at the same 4.410.
September 2026, ninfer_bench on NInfer's benchmark corpus, BF16 KV, greedy, three repetitions,
the default 120a build with each card's built-in device profile:
| RTX 5090 (550 W) | RTX PRO 6000 (600 W) | official artifact, RTX PRO 6000 | |
|---|---|---|---|
| prompt, 512 tokens | 21,622 tok/s | 21,705 tok/s | 18,040 tok/s |
| prompt, 4,096 tokens | 27,412 tok/s | 30,938 tok/s | 22,643 tok/s |
| decode, 128 tokens | 407 tok/s | 399 tok/s | 398 tok/s |
| decode with MTP (3 drafts, proposal head) | 541 tok/s |
The MTP row drafts over the benchmark corpus's prose (42% of drafts accepted); a short chat answer decodes at 742 tok/s on the RTX 5090. Decode reads about the same bytes as the official artifact, so it runs at the same speed; prompts gain from the FP4 tensor cores.
Apache 2.0, as both sources; see NOTICE.