The fastest inference engine for Qwen3.8-Flash-Next on Blackwell

Basalt (the engine) is MIT-licensed; the model files in this repo are derived from Qwen3.8-Flash-Next and stay under the Qwen Community License.

Model files for Basalt, a local inference engine for the Qwen3.8-Flash-Next mixture-of-experts model. Basalt targets one thing: Linux with an NVIDIA Blackwell GPU (sm_120). It runs on one RTX 5090, can use a second card as a peer expert tier, and keeps the experts that do not fit in VRAM in system RAM, so a large MoE model runs on a single workstation. It exposes an OpenAI- and Anthropic-compatible API.

Each .basalt file is self-contained: weights, tokenizer and chat template, MTP drafter, expert profile, 8-bit hyper-connection copies and the vision projector. This repository carries the .basalt packs, the conversion kit and the tuned drafter.

Speed

All numbers: 400 W GPU power limit, RTX 5090 + RTX 5060 Ti (peer expert tier) + 32 GB RAM, Ryzen 9 9950X.

Single stream, tokens per second:

Prefill at 64k context: 7,317 tokens/s (IQ3_XXS), 7,368 tokens/s (IQ3_S).

For comparison, Strata (the engine Basalt forked from) on the same machine, single stream: prose 166, structured 254, chat 205 tokens/s, prefill 4615 tokens/s.

Concurrency with --parallel, GSQ-RCO IQ3_S, total prose throughput:

Models

Quality measured with basalt-bench --quality: 20 chunks of wikitext-2 (512 tokens each), scored through the decode path against the Q8_0 reference. Lower KLD and higher top-1 agreement are closer to Q8_0.

The Unsloth Q2_K_XL, IQ3_XXS and Q3_K_XL rows are quality comparisons only; they are not Basalt packs.

The tuned drafter

The IQ3_S, IQ3_XXS and Q2_0 packs carry an MTP layer fine-tuned on the model's own output (self-distillation), which raises draft acceptance. Output is unchanged, because verify decides every token. Q4_K_XL and Q8_0 keep the original MTP layer.

The tuned layer is the mtp/dense section. Here it is convert/mtp/rt/dense.bin (116099072 bytes, SHA-256 dfe18647e0c95d05302907be88f021590119645a1567fcd770c8d309df852d75, described by convert/mtp/rt/dense-tuned.json), so a pack built from this kit with --mtp-source convert/mtp gets it. The original layer is kept as convert/mtp/rt/dense-original.bin; copy it over dense.bin to build an original-drafter pack.

Run

Requirements: Linux x86-64, an NVIDIA Blackwell GPU (compute capability 12.0), CUDA 13, enough system RAM for the experts that do not fit in VRAM. Build the engine from the Basalt repository, then put the settings in one basalt_config.json:

{
  "server": {
    "model": "Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt",
    "alias": "Qwen 3.8 FN",
    "gpu": "0,1",
    "host": "127.0.0.1",
    "port": 8080
  },
  "engine": {
    "expert-cache": "auto",
    "prefill": "auto",
    "spec": 7,
    "spec-min-p": 0.5,
    "max-context": 262144,
    "kv": "int8",
    "resident-experts": true,
    "vram-reserve-mib": 700
  }
}

bin/basalt serve --config basalt_config.json
bin/basalt serve --config basalt_config.json --parallel 4   # four requests at once

--parallel N (1 to 8) serves N requests at once in batch slots; --max-context is shared between them. The server listens on 127.0.0.1 by default; do not expose it beyond that without --api-key.

convert/: the conversion kit

The inputs tools/convert/convert.py takes besides the source GGUF, so any GGUF quant of this model can be turned into a .basalt file:

The two .gguf files of the kit are packed in convert/gguf-sources.tar (uncompressed, relative paths kept). Extract it at the repository root before running the converter:

tar -xf convert/gguf-sources.tar

python3 tools/convert/convert.py --gguf MODEL-00001-of-0000N.gguf --out MODEL.basalt --compat-bf16 \
    --mmproj convert/mmproj-Qwen3.8-Flash-Next-BF16.gguf --mtp-source convert/mtp \
    --profile convert/profile.bin --draft-vocab convert/draft_vocab.bin

Sources

• GSQ-RCO quants: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, quantized with GSQ and RCO by the Deep Algorithms and Systems Lab (IST Austria).
• UD quants and Q8_0: unsloth/Qwen3.8-Flash-Next-GGUF.
• The MTP drafter: the original BF16 MTP tensors from Qwen/Qwen3.8-Flash-Next, repacked by Basalt's converter; the dense part shipped here (convert/mtp/rt/dense.bin) is Basalt's self-distilled tune of it.
• The vision projector (mmproj-Qwen3.8-Flash-Next-BF16.gguf): from ISTA-DASLab (its GGUF metadata names ISTA DASLab as the quantizer).

The tensors are copied unchanged from these GGUFs; only the container is new. Basalt is a fork of Strata (MIT).

Citation

If you use the GSQ-RCO files, please cite the papers behind them:

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

License

All files here are derived from Qwen3.8-Flash-Next and are distributed under the Qwen Community License 1.0, whose copyright and permission notice is included in this repository and inside every .basalt file. In short: use, modification and redistribution are permitted; businesses offering Model as a Service or an AI coding/office assistant need a separate license from Qwen for commercial use; and products above 100M monthly users or US$20M monthly revenue must display the model name. Read the LICENSE for the binding terms. The Basalt engine itself is MIT.